State of GeoWebCache
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 351 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/69223 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Year | 2022 |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
00:00
Web 2.0Cache (computing)Core dumpSoftware developerPresentation of a groupSoftware maintenanceProjective planeComputer animation
00:19
Open sourceSoftwarePressureInterior (topology)GeometryServer (computing)Open sourcePresentation of a groupData storage deviceNumberStandard deviationProjective planeSoftware developerCollaborationismCore dumpLevel (video gaming)Variety (linguistics)Multiplication signOpen setOffice suiteGeometryInterior (topology)SoftwareComputer animation
01:00
Operations support systemClefGoogolOpen sourceSoftwareComputer fileInstallable File SystemChinese remainder theoremPolygon meshConfiguration spaceCache (computing)GeometryWeb 2.0Variety (linguistics)Point cloudCompact spaceJava appletMappingCommunications protocolRun time (program lifecycle phase)Computer configurationComputer fileOpen sourceConfiguration spaceSoftwareData storage deviceRepresentational state transferInterface (computing)Different (Kate Ryan album)TesselationTessellationGeneric programmingOcean currentOrder (biology)Game controllerDatabaseServer (computing)Electronic mailing listComputer programmingBasis <Mathematik>Library (computing)MiniDiscMereologySingle-precision floating-point formatSpeech synthesisMultiplicationLevel (video gaming)Multiplication signPoint (geometry)Set (mathematics)Abstract state machinesScaling (geometry)File systemLebesgue integrationProgramming languageComputer animation
04:02
Operations support systemTime zoneOpen sourceSoftwareMatrix (mathematics)Image resolutionPixelUniqueness quantificationVector spaceMaxima and minimaMacro (computer science)Computer fileSeries (mathematics)Greatest elementTesselationServer (computing)Multiplication signLevel (video gaming)Communications protocolWeb 2.0NumberMatrix (mathematics)String (computer science)Template (C++)Set (mathematics)TessellationStandard deviationVector spaceKey (cryptography)Uniform resource locatorAttribute grammarImplementationClient (computing)AreaInformationLink (knot theory)Time zoneCache (computing)Latent heatWordComputer reservations systemComputer architectureQuadrilateralClassical physicsUnified threat managementParameter (computer programming)Point (geometry)Elektronisches MarketingRepresentational state transferDimensional analysis1 (number)Software frameworkMereologyGeometryQuery languageMessage passingBitRaster graphicsCASE <Informatik>INTEGRALForm (programming)Revision controlRepresentation (politics)Computer configurationOpen setComputer fileInterface (computing)Descriptive statisticsMedical imagingMathematicsSinc functionComputer animationSource code
10:43
Operations support systemSoftwareOpen sourceDigital filterFiber bundleCache (computing)Compact spaceDisintegrationImplementationTesselationTessellationOpen sourceCache (computing)Electric generatorParallel portImplementationComputer configurationAreaNatural numberDefault (computer science)Web 2.0Software maintenanceBridging (networking)Software developerSet (mathematics)INTEGRALComputer fileServer (computing)Range (statistics)Multiplication signSingle-precision floating-point formatVariable (mathematics)Parameter (computer programming)Domain nameModule (mathematics)Integrated development environmentData storage deviceTable (information)String (computer science)Directory serviceConfiguration spaceMatrix (mathematics)DatabaseLatent heatOpen setTelecommunicationCovering spaceGroup actionCombinational logicMappingCompact spaceCASE <Informatik>Network topologyoutputNumberOverhead (computing)Interface (computing)MetadataGeometryMeta elementComputer animation
15:12
Physical systemComputer fileCache (computing)Open sourceSoftwareCohesion (computer science)AuthenticationArtificial neural networkTranslation memoryWitt algebraPauli exclusion principleMiniDiscRule of inferenceLimit (category theory)Server (computing)Maxima and minimaNumberComputing platformData storage deviceFile systemComputer fileLatent heatInterface (computing)NeuroinformatikSingle-precision floating-point formatTessellationInsertion lossCache (computing)TesselationMultiplicationFlow separationLimit (category theory)BitDynamical systemMiniDiscLebesgue integrationCASE <Informatik>Directory serviceMultiplication signComputer animation
17:11
Configuration spaceCache (computing)PixelMeta elementFaktorenanalyseFile formatComputer-generated imageryRead-only memoryDefault (computer science)Dimensional analysisVector spaceOpen sourceSoftwareSet (mathematics)Computer reservations systemServer (computing)Service (economics)Parameter (computer programming)Directed setDisintegrationInformation securityInheritance (object-oriented programming)Dependent and independent variablesServer (computing)Point (geometry)Configuration spaceTessellationCache (computing)INTEGRALAsynchronous Transfer ModeTesselationDirection (geometry)Parameter (computer programming)Doubling the cubeGeometrySource codeComputer animation
18:10
Configuration spaceStatisticsMiniDiscTask (computing)Zoom lensSoftwareOpen sourceData typeMilitary operationParameter (computer programming)Digital filterNumberMassRepresentational state transferConfiguration spaceProcess (computing)Matrix (mathematics)Game controllerData storage deviceCache (computing)TessellationTesselationAuditory maskingComputer animationSource code
18:30
SoftwareOpen sourceTerm (mathematics)Touch typingCache (computing)Process (computing)VolumenvisualisierungFlow separationMultiplication signBitRepresentation (politics)Row (database)Line (geometry)Keyboard shortcutServer (computing)Power (physics)TessellationWritingSoftware developerView (database)TesselationData compressionWeb 2.0Point (geometry)CalculationObject (grammar)Single-precision floating-point formatMaxima and minimaCASE <Informatik>Link (knot theory)GeometrySoftwareDefault (computer science)Level (video gaming)Representational state transferConfiguration spaceSoftware maintenanceWeightSinc functionProjective planeScalabilityElectronic mailing listTelecommunicationSoftware bugDirection (geometry)Computer animationSource code
Transcript: English(auto-generated)
00:01
Hello, welcome everyone, my name is Andrea Aime, as I said I'm a G-Server core developer but I also do contribute to G web cache and the co-writer of this presentation is Kevin Smith which is the current maintainer of the project.
00:21
I work for G solutions, we have offices in Italy and the US, the company is growing, last time I did this presentation it was saying 30 plus collaborators, now we are supporting 40 plus, we are supporting a number of open source projects with core involvement in their development, including geo server map store, geo node and geo network and we
00:44
offer a variety of services around them. G solutions is about openness so we strongly support open source, we are an OGC member because we believe in open standards and also support the standards critical to geo int.
01:02
So let's begin by asking ourselves what is geo web cache anyways? So geo web cache is first and foremost a tile server, it's a Java based stand alone tile server that supports a variety of tile protocols like WMTS, TMS, tile JSON,
01:23
X, YZ, Google maps, Bing maps and so on. But it's also a WMS server, not many people know that geo cache can be turned into a WMS, there is an option because it's an option away, it's a configuration option away which
01:40
is full WMS equals true in its configuration file and then geo cache can take tiles that it has in the cache, combine them, scale them and cut them in order to answer a generic WMS request. Currently it's working only on a single layer basis so unlike WMS you cannot list
02:00
like three layers and say give me a map of that, it works only if you are specifying a single layer but if anyone here or listening from remote is interested in sponsoring multi layer support we are very happy to talk with you or if you want to simply implement it and make a request, that's going to be very well received.
02:24
Geo web cache is based on a variety of tile sources such as WMS, MBTiles and compact cache and more, we are going to talk about each one of them in detail. Some of the tile sources are actually not tiles such as WMS so we need to actually
02:42
store the tiles somewhere and this is stored through an interface that we call the blob store and we have different options such as file system, cloud storage, SQL databases and so on. The configuration of standalone geo web cache is based on XML file but it also has a rest
03:02
API that you can use to add layers, remove layers, control grid sets, configure the disk quota and so on so you have an option to write your own XML file or use the rest API to control its configuration at run time.
03:21
If you are not much into the programmatic control then you can also use geo cache as part of geo server. So geo cache as I said is a standalone software and it was born that way but technological speaking it was so close to geo server using the same programming language, using
03:42
the same basic libraries that we actually managed to import it as a library and integrate it with geo server so that we can do point and click configuration and a few other interesting tricks that I'm going to explain later. So any time you download the geo server it actually comes with an embedded geo cache running inside of it.
04:03
So let's start from the top and work our way to the bottom, tile services. Geo cache was born as an implementation of TMS, the tile map service and eventually it also had the web map service cached protocol implementation.
04:21
These are the original tile services from OSGEO back from a time when OGC did not have a protocol for tiles yet, WMTS came later and the first implementation of geo cache was a TMS built on top of WMS layers so it was meant to cache and tile cache layers
04:45
and deliver them as TMS. Of course we do support WMTS web map tile service which is the OGC standard. The OGC standard comes in a couple of flavors.
05:00
One is KDP where you have these URLs with key value pairs in the query string that identify the layer, the X, Y and Z, the grid set and so on and this is kind of the first implementation that we made for WMTS but then we also implemented restful interfaces which has a slightly different approach and every parameter is actually baked into
05:25
the path of the URL and it doesn't have a query string instead. The capabilities document unlike other standards has explicit support for dimensions such as time and elevation or other custom dimensions so that your clients can discover which values
05:43
are available for time, elevation and other custom dimensions. Along with support for WMTS we have support for the 2D tile matrix set specification from OGC. That specification among the other things lists a bunch of well known tile matrix
06:04
sets that is definition of tile pyramids that have a name and a well known way of being organized. So we have names like web marketer quad, word CRS 84 quad, UTM some zone WGS 84 quad
06:22
and since we have the UTMs and also the UPSs that go with it to cover the entire planet we end up having a lot, a lot of built in CRSs that you can choose, sorry not CRSs, tile matrix sets that you can choose by just point and clicking without
06:42
having to configure stuff. If you're looking for your classic web marketer that all XYZ clients are respecting go to web marketer quad. If you want to make me happy choose anything else, like literally anything else.
07:01
In OGC 2D tile matrix set the Z parameter is a pure number. It's kind of an improvement compared to the older tile matrix sets. So in classic OGC, sorry not OGC, in classic geocache the tile matrix set and the tile matrix is identified by a long string and many XYZ clients do not like to have
07:26
a complex string as the Z, they only want a number. So in the definition of these tile matrix sets the Z is always a number which helps with interoperability with XYZ clients. Instead of having EPSG column 3857, column 0 instead of just 0.
07:47
In WMTS we have built in recent years a tile JSON integration. So when you go to the capabilities document of a RESTful WMTS it has a number of resources which are literally links. Sometimes they are templates where you have to fill in an XY and a Z and other
08:03
times they are just plain links. One of them in the capabilities document of GeoServer is a compliant tile JSON document. Tile JSON is interesting because it plays into the mapbox ecosystem, let's say, and you can point your mapbox client to the tile JSON and the tile JSON
08:24
describes templates for the tiles, the area covered by the tile set. If you are playing with vector tiles it describes which attributes are found into the vector tiles and so on so that the client can have full information about how to access the tiles.
08:43
As I said, typically OGC clients don't care but if you are playing into a mapbox client then this is useful. And then there is OGC API tiles. WMTS is going to eventually be replaced by this standard which is current in a draft form so it's not finalized, it's still subject to changes.
09:05
But it's a nice, modern protocol based on RESTful idea, open API descriptions, JSON representation of every resource plus optional HTML representation of every resource.
09:22
You can have OGC API tiles only if you are using the GeoServer integrated version of GeoWeb Cache because it has been implemented on top of the GeoServer framework for OGC APIs so it's not part of the standalone GeoWeb Cache but you can have it if you are using the integrated ones.
09:42
And it supports both so-called map tiles, that is images, PNG, JPEGs and whatever but also data tiles which in the case of GeoServer means mapbox vector tiles. Although with a little bit more implementation effort you could have also raster data tiles like tiny geotaves which then you could use in-client.
10:10
There are other services implemented in GeoCache like the Google Maps Compatible Tiles, the Bing Maps Compatible Tiles and KML Tiles. They were pretty popular once upon a time.
10:21
Now I don't see them used a lot but I think that the interesting message about all of this is that we have a number of service implementations and the GeoCache architecture is modular so you can come in and implement whatever other tile-based protocol you see fit.
10:44
Then let's move to the sources of tiles, the tile layers. The first tile layer was the WMS tile layer. It connects to a WMS server and generates tiles by making get maps
11:00
to the WMS server. Eventually based on configuration it might ask for a meta-tile, that is a group of X by Y tiles that is then downloaded and sliced into actual tiles. Meta-tiling is interesting because it reduces communication over it and because it typically improves labelling layout.
11:26
On top of that we have a number of options such as filter parameters. When you go to a WMS server like GeoServer you have a bunch of options attached to a particular layer such as default style
11:40
but also alternate styles and you might want to tile cache all of them and not just the default one and GeoCache will generate parallel tile caches for each and every style or you could have tile caches based on time, on elevation or SQL filter or environment variables and whatever other vendor parameter you can think of.
12:01
You go to GeoCache, you describe your vendor parameter, you describe the domain of that parameter that you want to cache. Like for example you might want to only cache a range of time or I don't know three possible values for an environment variables and not every random string that one could possibly input
12:21
and then GeoCache is going to generate caches for each and every combination of parameters. Of course you configure the set of supported tile matrix sets for each layer you might be caching in Web Mercator but also in WJS 84 but also in some Lambert
12:41
which is suitable for your area and so on and so on. Since we are fetching from a WMS which is not a natural tile source, we have to actually generate tiles and put them somewhere so this layer has to put tiles in a blob store. It has to put them somewhere and we are going to talk about possible implementations of blob stores later.
13:06
The ArcGIS compact cache is an open specification by Esri that allows to stick many tiles into a small set of files or a small directory tree and this has been contributed by external developers
13:23
so in the compact cache you go to your ArcGIS, you export your compact cache, it's fully seeded, it has all the possible tiles, you attach it to GeoCache and it's going to start serving tiles. So in this case we don't really need a blob store, all the tiles are already there.
13:40
We just need to figure out which tile matches the request and fetch it. This module has been contributed by an external developer and hasn't been maintained much so if you know anyone that can step up as the maintainer of this module or if you know anyone that would fund other developers to get a better sense of how this module works
14:01
and become a maintainer, please let us know. Another tile source is the MBTiles layer. MBTiles are single file tile caches based on SQLite so it's a little database with, I don't remember, it has one table for the tiles, one table for the metadata, stuff like that
14:23
and so basically all the tiles are stuck into this little SQLite database and again they are typically fully seeded so you attach this SQLite file to GeoCache and it's just going to query the right tile based on the request. Again, no need for a blob store, all the tiles are already there.
14:45
Under covers and something that people often do not know about is the GeoServer tile layer. It's the bridge between the layers in GeoServer and the tile cache and machinery of GeoCache. It's a custom implementation that makes use of a few things
15:04
like direct rendering engine integration. Okay, tile layer is an interface, you don't see what you like, you can implement your own tile layer. Blob stores, it's where we store tiles for WMS layers typically. You can store on the file system with a configurable layout.
15:22
We have a native layout which is sometimes a bit strange but it ensures that we never have too many files in our directory which on some file system can be an important feature or we can go XYZ and then generate a native cache that you can maybe export.
15:42
There is a number of blob storage integrations such as S3 but also Azure but also, whoopsie, let me go back, but also Swift. So generally speaking, we take blob storages
16:04
and store ties in buckets or whatever they are called in the specific platform and for example, we don't have right now one for Google storage. It would be nice if someone came and implemented it.
16:20
The SQLite blob store is another idea that we can use multiple SQLite files so it plays a little bit on MBTiles but it uses multiple separate SQLite files so that we can seed with greater efficiency tiles because we can parallelize the computation. Otherwise, all the inserts would accumulate on the single SQLite file and slow down stuff.
16:44
OK, blob store is an interface so you can go and implement your own. So Google storage, memcached are just a few ideas of what could be done. Also, when we are using a blob store, we can use this quota to limit the number of times stored on disk in case we are doing dynamic caching.
17:02
So you say, OK, my disk, I have 200 gigabytes and I will dedicate it to tile caching, no more and when I reach the limit, it's going to clean up less used tiles. GeoCache is well integrated with GeoServer with point-and-click configuration in every layer.
17:20
You can configure the grid set with point-and-click. You can do fancy stuff like the direct integration. The direct integration is that idea if our WMS request comes in, GeoCache will try to evaluate and see if it actually matches our cache tile and if it does, it's gonna just use the cache tile directly. Otherwise, if it's cacheable, it's gonna cache it.
17:43
If it's not cacheable because of some request parameters that are outside of the definition of the caching, then it's gonna do a dynamic request. So it allows you to dynamically switch between caching and non-caching modes based on your current request. We have also automatic cleanups. So when you edit a style or you do WFST on data,
18:06
GeoServer will know which tiles to drop and it's gonna automatically clean your cache. There's a REST interface, as I said, for full configuration of all the layers, the blob stores, the tile metrics, this quota and also REST API for seeding and cleaning.
18:22
So creating tiles as a mask job or cleaning tiles as a mask job and you can programmatically control this. In terms of community, please get in touch with us. You can download the packages from GitHub,
18:40
from SourceForge and there's a link to the documentation. In terms of communication, we have a user and a developer main list where you can drop and ask questions or ask about your potential participation to the project. And in terms of development, well, throw us a pull request for new features,
19:03
bug fixes or maintenance, documentation, place documentation, it's always less loved. And also testing, when we are about to release, please try out our release candidate and let us know if you find any issue. And with this, I'm done.
19:27
Thanks, Andrea, that's a very powerful software. If you can ask questions on Venueless, we have a record of them. So there's one question online so far and then we can throw it out to the floor.
19:40
So it's a two-part question, Andrea. Is it better to have separate geo web cache that can scale beside a geo server instead of using the embedded one when working with clustered infrastructure? And how easy is it to bind this kind of geo web cache with geo server to have a fully scalable architecture? Okay, so it depends. It depends on where you want to go,
20:02
how much you want to scale and how many tiles you want to build. So if your objective is to, so let's say that you have a large installation with many nodes and you have a geo web cache, in that case, in front, and you want to do a larger seed job, then that single geo web cache
20:20
can literally hit the entire cluster and multiply the speed at which the seeding happens by leveraging the full cluster. So it's definitely going to speed up intensive seed jobs a lot. On the other end, since it's external, we will have to make a map, compress it to PNG
20:41
and then decode the PNG on the other side. So on the single request, it's actually going to pay some extra time with the integrated one, render the map and save it as a tile directly. So there's this kind of trade-off. If you are going towards pre-seeding a lot and you want to have a maximum performance,
21:02
then yes, having a geo web cache in front separate is better from a performance and scalability point of view. If instead you are going to do a lot of on-demand tile caching where the tile cache starts empty and then as the request comes in, we generate the tiles on the fly, then the traffic is typically going to be lower
21:23
and it pays off to not have to compress PNG and the compress PNG over the wire. So I would prefer a embedded situation instead. In terms of configuration, the embedded one configures itself
21:40
as you add layers by default. So it's really, really easy. Maybe sometimes too easy, like you end up with so many tile cached layers that you don't even realize. The standalone one needs you to write XML configuration for the tile layers, for every tile layer that you want to control
22:01
and you can use the REST API to automate that. So it's feasible, but it's going to be more work, definitely. Okay, great, thank you. A second question has come in. If the, I presume it's the OGC tiles API will be in GeoServer, what role will GeoWebCache have in the future? Well, it depends.
22:21
We could go and implement the tiles API also in GeoWebCache, maybe in a lighter way, maybe like, for example, only with the JSON bindings and forget about the other bits like HTML representation of resources because they are not mandatory and implementing just JSON is like pretty, pretty quick.
22:44
So that would be a way. Another thing that I'm thinking about is that we could also flip it and say, okay, let's say that we are always gonna use the integrated one, but we come up with a clustering solution that if we start a seed job on one node,
23:01
spreads out the calculation on all of the others that are in the same cluster. That's also feasible and would retain the point and click support and all the fancy extra abilities that GeoServer has. Okay, great, thank you. Are there any questions from the room?