MapCache: Overview of MapServer's tile caching server

Video in TIB AV-Portal: MapCache: Overview of MapServer's tile caching server

Formal Metadata

Title
MapCache: Overview of MapServer's tile caching server
Title of Series
Author
License
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
2014
Language
English
Producer
Foss4G
Open Source Geospatial Foundation (OSGeo)
Production Year
2014
Production Place
Portland, Oregon, United States of America

Content Metadata

Subject Area
Abstract
MapCache is the MapServer project's implementation of a tile caching server. It aims to be simple to install and configure, to be (very) fast (written in C and running as a native module under apache or nginx, or as a standalone fastcgi instance), and to be capable (services WMTS, googlemaps, virtualearth, KML, TMS, WMS). When acting as a WMS server, it will also respond to untiled GetMap requests, by dynamically merging multiple layers into a single image, and multiple tiles into an arbitrary image size. Multiple cache backends are included, allowing tiles to be stored and retrieved from file based databases (sqlite, mbtiles, berkeley-db), memcached instances, cloud REST containers (S3, Azure, Google Cloud Storage), or even directly from tiled TIFF files. Support of dimensions allows storing multiple versions of a tileset (e.g. one per customer), and time based requests can be dynamically served by interpreting and reassembling entries matching the requested time interval. MapCache can also be used to transparently speedup existing WMS instances, by intercepting getmap requests that can be served by tiles, and proxying all other requests to the original WMS server. Along with an overview of MapCache's functionalities, this presentation will also address real-world usecases and recommended configurations.
Keywords mapserver tiles wms wmts mapcache tms mbtiles
Loading...
Medical imaging Email Pixel Connectivity (graph theory) Multiplication sign Tessellation
Module (mathematics) Server (computing) Standard deviation Multiplication Mapping Computer-generated imagery Multiplication sign Image resolution Debugger Projective plane Instance (computer science) Client (computing) Web 2.0 Heegaard splitting Data management Different (Kate Ryan album) Personal digital assistant Extension (kinesiology) Communications protocol Library (computing)
Scripting language Machine code Mapping Demo (music) Code INTEGRAL Software developer Interface (computing) Connectivity (graph theory) Student's t-test Medical imaging Energy level Communications protocol
Dependent and independent variables Server (computing) Dialect Key (cryptography) Wrapper (data mining) Multiplication sign Chemical equation Source code Debugger Set (mathematics) Representational state transfer Stack (abstract data type) Computer Rule of inference Tessellation Medical imaging Cache (computing) Digital photography Googol Personal digital assistant Proxy server Communications protocol Asynchronous Transfer Mode
Server (computing) Email Meta element Multiplication sign Source code Mathematical analysis Set (mathematics) Web browser Client (computing) Regulärer Ausdruck <Textverarbeitung> Code Formal language Tessellation Revision control Cache (computing) Medical imaging Message passing Arrow of time Family Traffic reporting
Metre Medical imaging Image resolution Multiplication sign 1 (number) Energy level Representational state transfer Data compression Tessellation
Matching (graph theory) Mapping File format Weight Data storage device Bit Graph coloring Web 2.0 Medical imaging Word Geometric quantization Color space Energy level Data compression Resultant
Overlay-Netz Satellite Tesselation Multiplication sign Data storage device Bit Client (computing) Mereology Graph coloring Tessellation Band matrix Medical imaging Strategy game Personal digital assistant
Cache (computing) Type theory Multiplication Standard deviation Tesselation Set (mathematics) Open set Mass Power (physics) Engineering physics
Group action Arm Image resolution Multiplication sign Software developer Zoom lens Data storage device Mereology Limit (category theory) Medical imaging Cache (computing) Operator (mathematics) File system Energy level Software testing Extension (kinesiology) Form (programming)
Satellite Presentation of a group Hoax Multiplication sign Source code Set (mathematics) Insertion loss Client (computing) Mereology Dimensional analysis Subset Medical imaging Hooking Computer configuration Single-precision floating-point format File system Data conversion Extension (kinesiology) Area Mapping Block (periodic table) Structural load Data storage device 3 (number) Representational state transfer Instance (computer science) Variable (mathematics) Regulärer Ausdruck <Textverarbeitung> Demoscene Tessellation Orbit Band matrix Message passing Data management Photographic mosaic Process (computing) Vector space Internet service provider MiniDisc Reading (process) Spacetime Asynchronous Transfer Mode Geometry Point (geometry) Statistics Server (computing) Backup Momentum Divisor .NET Framework Number Template (C++) Revision control Population density Term (mathematics) Operator (mathematics) Ideal (ethics) Authorization Energy level Booting Task (computing) Default (computer science) Multiplication Focus (optics) Interface (computing) Polygon Expression Database Directory service Limit (category theory) Timestamp Cache (computing) Database normalization Personal digital assistant Communications protocol Library (computing)
Hierarchy Energy level Quicksort Data compression
Server (computing) Tesselation Code Multiplication sign View (database) Tessellation Bit rate Personal digital assistant Computer configuration Mixed reality Cuboid Data conversion Remote procedure call Proxy server Error message Geometry
Multiplication Computer configuration Personal digital assistant
Thread (computing) Computer configuration Endliche Modelltheorie Front and back ends Tessellation
mn it having a component that's been added to maps of in 2 years ago officially I think it's still of 3 million e-mail in just a short introduction about what what's tidings so tiles that actually precomputed usually
256 by 256 pixel images that allow fast access to static the static data that's been as being cash someone for a rapid access at a time is aligned to 100 which
is basically a subdivision of the world along a given for a given projection for a given extent spend across different resolutions and files contain data that has been rendered by some third-party service and and usual case another WMS server all maps of itself for cool data also whatever I once a time of being generated by the split up by the times of installed in a cash back and so will be looking at all the available cash Watkins we have a available and then that's how can be self back to the client so over the web the respecting difference services standards yeah the not cash itself is actually a tiling library more answering server and on and it has a front ends let's try either to the Apache web server as Fast CGI instance as the native engine next module
or as a native mobile and it's versatile in the sense that supports multiple cash likens it supports multiple client protocols and has advanced time management features which allow you to pre
generate uh seeds beforehand so this will here we compress images to getting integrates interpolated such data that's not present from lower levels it's written in native code C which is a fast and mean and that doesn't a tiny demo interface that allows you to quickly set up a service and and just copy pasted to a script code this needed to get that you stats solidarity in your your development and crinkled and in the code is now 4 years old I started in 2010 and it was officially integrated as maps of the
components the maps of the student in 2012 so often look about the protocols it's our and implemented and how you can request of America so it's in user's either
standard xyz addressing the came soup overviews too few times in google the key and it has also a WMS wrapper to create them to read to me and from the titles of going to movies of so for standard time addressing it supports the TMS protocol which is the you always due to protocol for addressing tiles it supports WGS in either RESTful RESTful key value pair modes which is basically very soon it assumes that the affected by the UGC suppose addressing by virtue of quot piece not addressing and the standard xyz addressing it's also supports a WMS did not request which means that is full computes uh and we get not a request for a given size of layers and use the tiles in 2 thousand cash instead of using the native data itself or in most cases this is usually much faster than hitting it obvious itself because you're using precomputed image tiles of the source so it response to untiled requests by ascending those styles from the the caches it can either assembled them vertically in the sense that if you are asking for multiple layers that little stack multiple times 1 and the other fusion them into a single time and also horizontally which means that if you ask for a 1 thousand by 1 thousand images or stack photos once 1 next to each other decrees that image and it also acts as a kind of uh but OTC imbalances is because that's the main usage as a proxy for the
services so in that case it can be the front end to your services it'll intercepts the requests that that tiled all that can be served from dial caches and then you have a set of rules that can allow you to forward other requests to other servers so you
have an expressive language with the regular expressions this means if you can send WFS request to a WFS of the you can send WF 1st request of version 1 . 1 2 1 7 and 2 . 0 to another server you can also serve static that data from the from another analysis and miscellaneous features which just the together here and it creates the caching HTTP headers so the clients don't keep requesting over and over the again for the same time uh if we allow it to to live that long and their brothers browsers cache it's can expire more Nestle's radically so basically you if you define that's a time in a cache can only be 1 hour old it's will automatically delete that if it's requested after 1 hour recreated back from the source
it'll report arrows idea and as a message I and that's the mentee image and all always just basic status codes inside there you know it should be requests and the support for meta timing so that's creating multiple tiles from a single large get requests to the source that we have a set of you could watermark
that tiles with your local with you that's what you want and that's um and it's can also the and tiles from lower levels of metals so basically if you have at 10 meter and the rest images that you want to serve them up to up to some level 2004 1 for wraps there's no use caching those uh no resolution tiles useful just retrieving from the middle Rosicrucian data that's that's available and once you image comes back from the WS it can be recompressed an optimized before being sold into the town cash itself button this means that this is useful if you using roster data and and that you want to avoid the double J. peg compression-decompression 1 ones from the WMS someone want from this that I'll catch so you'd request PNG from missiles WMS and and then compressed to get big only once when a story to get our cash can also allow you to be aggressive on the compression levels you're applying to the times when a storing them the new you
preceding everything or you can also choose to use a letter and a less aggressive compression for doing on the mound and compression of image tiles so let's see if you use the but at the cost of a
higher weight images and it supports multiple image formats that can be stored or return to their clients and so the 2 basic image formats used to on the web so PNG on which you can play with the compression level that's applied to the painted and you can also apply quantization to the that data to store all those stars as a bits by choosing the correct 256 colors that best match the image that's that's being stored for debate can also play on the compression quality level and you can choose in which color space the the due the compression takes place usually YC meteoric gets you better compression results there's also word
kind of fun feature which allows you to choose that map cash and anarchy choose in which format is going to store it styles it's going to store J. back if it sees that the image data is fully opaque and then Djindji if it sees this this transparency and so basically
that means if you have a satellite roster that doesn't cover the whole world and all the parts that are fully covered by the US it will be stored in Japan to reduce the bandwidth and the images on the side which have no data values will be stored and returned as PNG so you can overlay that's where the other layers on the client we and it's also try to woo intelligently handle empty uniform tiles depending on the caching back that I use it so use different strategies once it sees that the tile contains no date so just 1 single color the caching under this it's all users symbolic linking to world war 2 1 2 1 file as can also for cash back and just
store the color and the cash instead of storing the image style and then this dynamically recreated PNG image on the fly at the request time returning a 1 bit PNG of a uniform color so I talk from exactly the size of that and you find something around 180 bytes but if you are in the case where you know your cash is already 40 seeded
and it so treats a tile that's not present in the cache fully transparent find return that the it and so now we talk about initial mass not cash supports multiple grids not only in the global market 1 so you can define multiple grids power and for some datasets US settings they can serve them up in WGS 84 for for opening on the open market or but it's also automatically detects what you can configure to respond to greater aliases so that the eternal AI problem between the against each we've been pursued 57 and
needy pretreated 990 and touching 1 is handled this can handle non standard as well non EPS goes in that case but inside the grid you can also configure type set to only be configured for certain
extent of certain resolutions of regret so just say I'm using the Google group but some only want the 1st 2 levels and then 1 that 1 day if you want to change that's and and a switch to a higher zoom level you just have to change that's and not have to recreate although styles seen for the restricted extends single can only going to cover the USA um and then switch to the whole world and not have to recreate all the times you've already created through the the cash back which is basically the most important part of the of the cash and so a gaseous back and it's able to store a file data for a given x y z so that's the basic part of its arms to be able to become a cash you back and has to support of 4 basic operations which is there exists in a with a title it is in the cache given the given x y z get back image data for a given xyz store data form that's 1 and data and there are some specific Cox so all features that are only available for certain Catholic and so not going to more details about each cash back and that this gaseous thought
trial as directly in the file system and that I put here mainly for development testing or small type sets because storing millions or billions of stars on the file system you use the very quickly gets into the limits of what the file systems are designed to do and so the the prose about this discussion
it's very simple to set up he just give it a directory not careful source doesn't it's it's relatively or very fast so it can support detection of blank files that kind of and you have the option to store titles given different layout so you can reuse existing gaseous that you have available from other service uh so it's by default is going to use the layout used by the what's the catch created which can also read caches created by GIS uh well when you can sort by your own templates is to store on given given explained and the economists that it's difficult to manage that large number of files you it's difficult to get statistics about how much space is occupied by a both finds it difficult to copy them from 1 place to the other and you hit file system limits depending on the file systems of course but still in the usual case you run out of i know it's you run into too many files per directory weather and that you may also wastes I'm source space are given the file system block size so just a block size of 4 12 kilobytes administering 128 by tiled and you still have 4 kilobytes that occupied by that time it supports space like caches so this is just storing and that the final data as the blob in SQL database with the x y and z columns so the prose about this and back and it's it's it's a single file is going to contain all your time cash easy to copy move over to another server but on you can extend that to support spangly schema you want what supposing it stores data the problem you have xyz columns button so if you have existing issue files with thousands and you can just get map cash into it and its orbit in the from its and and is efficient in the disk space in a sense that's the only the space actually the occupied by the time is going to be taken on the front and the comes about this foot back and it's that so you may need some tweaking to do pass advance variables to askyou like when you creating databases when you have more than 1 terabyte of these are not remember was the exact limits of floor for that 1 that's around wonder whether I think and then another problem is that so as you like this isn't designed to handle multiple insertions concurrently which means if you have multiple map cash instances that are trying to push tiles inside the cash there's a knock on the database and you slowdown insertions very noticeably it supports a third-party memcached serve as a back and so on and uh memcached is there for storing transient data they won't resist reboots ideal for temporary data which for forecast for sensors whatever they is only valid for a limited amount of time what can be easily be computed and the prose of this these cash by Kansas that's you can distribute the load between you know all different that caches instances and then cache server and casual do automatic pruning for you of your caches or it's the details that haven't been useless and I will get pushed out of the cash to leave more room for the new terms coming all the cons is that as it's memory-based stories that you have limited storage available in this year very deep pockets from lots of RAM it Judith caches up by and cashed that use only and for us the data in the back it's specialized to store satellite imagery so it's going to store the image does directly inside to file and not actual read inside this to fight for the for the encoded data you can choose to store a given number of styles part the final so if you storing 2008 2007 4 million tons inside 1 2 5 and reducing the number of files you have just on the file system by that factor so and some it allows you to I the creator steadfast by by a third-party service as long as it's to the grids using around and to determine and it greatly reduces the number of fires you have just on the file system and the limitations of this back and this that it's uh limited to get data so you can store images with transparency European PNG data only do but that's a limitation of the find itself and you can try to cut concurrently to a single TIFF files so there's a big block around the writing operations so you know to processes right to this interface and that's if doing updates of the lesions inside that differ then library won't reclaim the space that you that you've removed from title should doing lots of updates and that's not the the former to use you to find work and continue to grow and grow and this is the new cash decided to the version that's coming out so of the rest caches to stall tiles on 3rd parties server that supports rest protocol so it's simple get put and on the on the URI and by itself this probably wouldn't be very useful but it's uh and it comes in handy because they are all authorization hoax to hook into popular cal search providers so you can if you have a club uh credentials you can still touch
to S 3 Microsoft Azure and Google account storage with us and once just storing thousand that's the cost-benefit ratio to to take into account of course because so it's not free on those services but then you have to understand that you usually have to run your not cash instance inside the same infrastructure as the caches themselves I given that's your bandwidth costs to take into account so you don't want to pay the bandwidth between you know as free storage and yourself and it just lets it so on and unless are of course unless you have a fully seated cash in which case you can just print them to the Urals of your your rest points and that's the yeah usually it's needed with the with the net cash instances running on G to it and there's a c that's the and that's shipped with net cash so the process of seeding is just the pre generating times and at 1 point so they are available at high performance of until then after that and so this you there's a multithreaded all multi-process which means you can have multiple instances running at the same time to if you to take advantage of of multiple WMS serve a multiple-chip use on the cash instances and it works in drill-down mode which means to take advantage of the file system caches on the WMS server so instead of just looking through the tiles by x lazy and naively it'll drill down so you always staying in the same area of data on the WMS so you the speed things up well known you can see on a subsets of your tireless sets if that's what you want but only for a given dimension you can regenerate old tiles that are older than a given date if you know that's uh of from 1 dates back was you you need to regenerate you can restrict to the resume levels you want to see the the extent and you can also restrict to arbitrary geometries so with the OGI expressions in text to extract the geometry so is that you can say OK I want to see level 18 on the polygons where my population density is is bigger than a given value to pre-generate where people actually are going to be using and there's also a pruning mode which will allow you to details that's also match the given criteria a and I talk about dimensions for this he does just that I'm going to talk about dimensions so basically fertile said you can store multiple versions of the of the image data that typically if you think of forecast data then and you will have a date for forecasts and you want to store the times for a given date or for forecast actually have 2 dates yesterday's when the focus has been made in the data for which the forecast is that it so this is supported you could uh the typical dimension could be a client idea if you need to create time sets that are different in restricted by extends for a given client ID or for elevation for temperature maps was and in dimensions can be expressed as the allowed values at intervals typically for that would be for innovation or just a regular expression this was that mismatch we've also added support for time dimension and in the case where you have an external database that provides you with the available time stamps that uh you can serve you can either Bustamante and interval so start time and end times in cash and it'll assemble the tiles that correspond to that interval so that can either be done horizontally if you make it fit in the case of satellite images where you have uh multiple scenes and you want to create a mosaic of all the images that have been created in 2013 made for example but can also be what's coming next is an animation so you could animate also with treating an animated gif of of your time over time but for that to what you must supply a database of the available time stamps because what you get as an impetus just an interval so you have to know which of the individual values that correspond to that and in future work that's known as King's it's and the pipes for for of conversions but maybe native could all sources so we don't have to go through WMS services and just read directly from the from the roster data an advance cash management so that would be the fade over caches redundancy if title isn't present in the cache and look into another 1 all in all it's just like and failed and use of backup coach back and and or to be able to move cash and tiles from 1 cash 1 slow cash to a faster 1 to from file sketches to non-cash and the last point also that needs to be addressed rather rapidly is being real close store more than just a image tiles which vector times so either you just grids on upcoming vector task and I'm over now so if you have any questions at books at
an act to creations that 1st minus about they're due to catch this is just sort of dollar do and command they get like that of all the user to follow it it doesn't support over the years from now it's only supports so you'd have to create 1 the hierarchy by level level and it's uh it's it's you created title text so if you pass the unsealed titled equals yes and compression equals J. that's when it's going to be used as a container and the 2nd 1 was about the In this
Cedar in annoy you start time catch the knows of all them proxy and establish for quality problem thus from the cresting at tile from remote luminous it in handling any errors in asthma proxy you could have a conversion rate would lead to the we crest again when that failed as anything available in the college button uh I think there's some code that wanting to be able to resume seeding after an hour so you can
start off from where the error happened in so was so little fail in that case and once you fix a WMS server the view to restart from where stopped OK thank you and thank you I see denied geometry I will use the bounding box the geometry your actual clipping of tiles which would their sector the x 250 with it and another 1 have vertical assembly what mixing options are available displayed
on-screen multiplies Servedio I only screen it yeah adding adding other composite options what makes sense and yeah of the
Cedars multithreaded but which
back end you recommend for doing multithreaded seeding since a lot of to walk on the back and so a well not this cue lights in in case of and
men get would probably work well in that case and and the world well I'd like to investigate into it modifying the he this so you can push tiles to the back end in a single thread so you have the more you want models that you're right all that could be another option also it
the any other questions the thank you
Loading...
Feedback

Timings

  514 ms - page object

Version

AV-Portal 3.21.3 (19e43a18c8aa08bcbdf3e35b975c18acb737c630)
hidden