We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Managing update of tiles of dynamic data

00:00

Formal Metadata

Title
Managing update of tiles of dynamic data
Title of Series
Number of Parts
95
Author
License
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Identifiers
Publisher
Release Date
Language
Production PlaceNottingham

Content Metadata

Subject Area
Genre
Abstract
Tiling is currently the best solution to achieve high performance and throughput for serving map images. However, because tile images have to be prepared, tiling is often only used for relatively static data. The Dutch national facility for presenting governmental spatial plans, www.ruimtelijkeplannen.nl, is used intensively (app. 15.000 plans) and has high performance demands and therefore wanted to make use of tiling. Because plans often change and are added and deleted from the central database, a special solution was developed to manage the daily update of tiles. The presentation will concentrate on the solution to manage the daily update of tiles, the Tiling Manager. The Tiling Manager software queries the audit trail of plan updates, executes tiling tasks in collaboration with GeoWebCache and monitors progress. We had to deal with several challenges to realize the requirements, such as monitoring progress of tiling tasks in GeoWebCache, run-time generation of GeoWebCache configuration, optimizing WMS performance and assuring that the services will never present old tiles if new plans are available. In addition to the solutions to these challenges the presentation will show the technical architecture of the Tiling Manager.
AerodynamicsBoom (sailing)World Wide Web ConsortiumLevel (video gaming)Zoom lensMereologyFluid staticsSymbolic dynamicsTask (computing)Interface (computing)Line (geometry)Configuration spaceTemplate (C++)NumberAreaZoom lensBasis <Mathematik>View (database)Multiplication signTesselationPublic domainDatabaseCountingData managementPhysical systemService (economics)Total S.A.Web pagePlanningLevel (video gaming)Moment (mathematics)InformationComputer programStress (mechanics)WhiteboardExtension (kinesiology)Centralizer and normalizerTask (computing)Workstation <Musikinstrument>Client (computing)Projective planeFreewareNetwork topologyCASE <Informatik>Instance (computer science)Natural numberAddress spaceCollaborationismArithmetic meanComputer hardwareDynamical systemOffice suiteConfiguration spaceRepresentational state transferTessellationArithmetic progressionCache (computing)Web 2.0Computer fileDot productGoodness of fitEstimatorWebsiteMereologySoftwareSoftware developerServer (computing)GeometryQuery languageDifferent (Kate Ryan album)Template (C++)Computer animation
Cache (computing)BitComputer architectureSoftwareUniform resource nameOvalDatabaseMereologyConfiguration spaceEmulationMUDTessellationDegree (graph theory)Link (knot theory)MultiplicationMultiplication signType theoryMereologyTable (information)Cycle (graph theory)Data conversionLine (geometry)InformationElectronic mailing listLevel (video gaming)Data managementArithmetic meanINTEGRALTesselationObject (grammar)PlanningObservational studyNumberWeb pageHuman migrationWhiteboardDatabaseInstance (computer science)Computer programBarrelled spaceAdditionBitRow (database)TelecommunicationTask (computing)Computer configurationOrder (biology)Service (economics)MathematicsInsertion lossTemplate (C++)Zoom lensWebsiteView (database)Process (computing)Flow separationData transmissionMoment (mathematics)Incidence algebra2 (number)Link (knot theory)System callCuboidWeb 2.0Numbering schemeTessellationSocial classConfiguration spaceCache (computing)InternetworkingTransformation (genetics)Proper mapComputer architectureSoftwarePresentation of a groupGeometryAdaptive behaviorArithmetic progressionPerspective (visual)Computer fileCategory of beingServer (computing)Office suiteConnected spacePasswordComputer animation
Presentation of a groupTessellationDegree (graph theory)Link (knot theory)Multiplication signLink (knot theory)Computer animation
Transcript: English(auto-generated)
I'll try to make it, keep it brief. Good afternoon everybody. Just after 12. My name is Rob Stoecklenburg. I'm working for E-DAISH in the Netherlands, a software developer.
E-DAISH is a small company in the Netherlands, a geo company that has clients among mainly the governmental organizations, provinces, councils, police and we were once involved in a project, a national project where we had to use tiling but where the map data was updated very regularly
and we had to meet and solve some challenges in that respect and develop special software to do what we wanted and to keep up with the goals of the program. The, oh I'm going much too fast, sorry. Okay. The Dutch government has a national facility
for showing spatial plans at all levels, council, governmental, provincial, all kinds of spatial plans that are useful for public use and professional use and at the moment we have
around 15,000 plans in the system and it's growing daily. It's really taking off. It's publicly available through a website and besides the website there are facilities for professional use as well. You see every, we have 15,000 plans, every dot is a plan and it's almost the
view of the Netherlands is coming out through all the small dots. Users have to be able to zoom in to plans, to pan and to zoom out. So we need a good performance and that was one of the
demands that the project had. I show you an example, a plan more in detail from the hometown of our company Raisin in the east of the Netherlands and the website gives wonderful
means of searching. I just found this plan in our town by typing in the address of our company's headquarters and it's a small plan that says something about how to develop the
center of this town. All the documents that belong to it, all the official documents can be found as and this is a really common local plan so to say. It's difficult to translate the specific items
into English but you can say this is a local plan and each local plan has about four to five map layers including borders, including an overview layer and these five layers are
used for the majority of plans in this facility and they all constitute the whole service area of the Netherlands. Maybe a few percent of the total number of plans is different, they have often a larger service area, they are more in the national area or the provincial areas of
special plans and they have their own specific layers with them and that makes one of the challenges in this case. The topography, the map of the town and surroundings that you see
is pre-tiled because it changes maybe once a year or even less so we do that on separate hardware. In this spatial plans project plans are updated daily because councils and everybody
who is using it from the government really wants to be able to present the new data as soon as it is ready so we have from a few plans a day to a few hundred plans a day per day that are updated in the system and have to be shown on the map as soon as possible. We provide
for a number of zoom levels from the overview of the Netherlands up to the utmost detail of a certain plan and we need to do that for different map layers as well. Most plans contain several layers like the plan I just showed before it contains five layers
and some of these special plans contain 10 or more layers as well and we have to provide for in total we have 700 map layers in the system that we have to provide to users.
To give you an example of how many tiles that could be for the topography layer I have been counting the number of tiles for the whole of the Netherlands for all these zoom levels and that's for one layer only five and a half million tiles and we have 700 layers in total
and this means that if we would not use tiling it would result in high demand on the WMS because at the daytime the map server is also used by professionals as well and
it serves the map data for the website. So we had to come with the solution and we had to use tiling to diminish the demand on the services but what we only wanted to do is tile
parts of the layer that were changed. We can't change the whole layers every time and the customer also had said that they wanted to be ready with tiling before office hours so there would not be any stress on the WMS during office hours.
We came up with a solution that we call the tiling manager. Static data is pre-tiled like topography and dynamic data which can change per day is tiled before office hours. We use the web cache which is a perfectly sound solution for for tiling for static tiling that is
so in this case we're not smiling yet we do need some something extra and that is the tiling manager which sits between it sits between the planned database of the whole project and the web cache and it queries the planned database on a daily basis
and then makes tasks that it can issue to geoweb cache and we had to change to adapt geoweb cache as well to be able to communicate it with it in in a way that we wanted and to to be able to monitor the progress of geoweb cache. We faced a number of challenges as I said the
monitoring of geoweb cache was wanted and we also wanted to be able to to stop geoweb cache because the tiling manager can get a command to stop everything it is doing at that moment so it has
to stop the geoweb cache instance as well with the tasks at hand and we had to extend the rest interface of geoweb cache. The tiling manager itself has a simple information page
which shows the plans at hand all the layers per plan the zoom level to which the layers have to be tiled and during tiling it gives the percentage of tiles ready and also an estimate of the time still to to expect the tiling will will need to to go on and because we have several plans
that that have their own their own layers the geoweb cache has a configuration file with the all the map layers inside and we only have five of these common layers that are common for a
lot of plans but there are some plans that use specific layers which we can't predict beforehand so we have to generate a configuration for geoweb cache using a template and the tiling manager knows about the plans it knows the template it fills in all the details
and it makes a proper configuration for geoweb cache that it needs we also did some work on optimizing the wms performance mainly by by using some extra caching so the the database requests were diminished the number of them and we
also found it very handy to be able to shrink the pngs that were produced 24-bit pngs to 8-bit using png quant which is a a standalone program that we use for every child
and the transformation is also built into geoweb cache and that means that the the i think the size of these tiles is indeed going from 100 to 50 or 30 percent size so it's really helpful to get less transport over the over the internet
and what was that was that we had up-to-date tiles so when a council changes a plan today and they upload it to the site they really want the map and the view in the next day
to be in order we can't have old tiles running around or or still being there so up-to-date tiles were was very important and this was ensured by the cycle that we made
we built into this tiling manager we have the plan database and the tiling manager takes all the plans that were updated deleted or or added it finds all the layers in all these plans it makes
for a task for each layer a geoweb cache task and all these tasks are visible in the information page of the tiling manager and the cycle starts with generating a new configuration so for all the plans that we have in in the morning so to say we generate a new configuration for geoweb cache
up to date then we're going to truncate to delete all the tiles of the changed plans of the of the of the the day before so we erase the old situation delete all the tiles that that were changed
next we send the new configuration that we generated here we send it to geoweb cache and we reset geoweb cache because we have to make sure that geoweb cache can read this new configuration file this was the most safe option we had then after that and
geoweb cache is back online we give it seat tasks so it makes the tiles based on the new configuration and it makes tiles which represent the new plans and in the map later on
and all these tasks in this info page are being done bit by bit until the list is empty and then we're ready for the day and hopefully before office hours a small idea
about the architecture there is one tiling manager and there can be multiple geoweb cache instances and they can each run in they run in a tomcat and they can run all in the same tomcat
or in a separate tomcat or on separate servers communication is via http so there is a flexible way that you can deploy the tiling manager and the geoweb cache instances you can have many geoweb cache instances if you want but they all do get the same configuration
and the same configuration means that they all send their requests to the same wms to get and produce the tiles the tiling manager is trying to find when it goes through its list through the cycle and through the list of tasks it finds a geoweb cache that is
doing nothing at the moment and send a task to it so it can do the tiles for that plan for that layer and then finds the next geoweb cache that is doing nothing and send another layer to that one and so forth and if they're all busy it just waits and monitors progress and waits until one
of these geoweb caches is free or the list of tasks is is empty and all that is configured in a properties file for the tiling manager it's a bit bad well it's not readable really but
from a software perspective there is a tiling manager class that accepts http requests from users translates that into calls to a manager service and the manager service has a pool of
that holds w no geoweb cache instances in the pool we have several geoweb caches representing geoweb caches and and the status of geoweb caches is also monitored in in a special class in a special object all is written in in java
and i think that is that if i show a very short overview of the database scheme there is a table that that contains the the task the cycle of of the day so to say a row for the cycle
of the day there is a big table that contains all the status items that are important for the for the task at hand the the plan the layer type the the zoom level we need to go to answer etc etc then there is a table that combines the type of plan with the layers that the plan
is normally normally containing and we have for the pre-tiling for for example when we want to pre-trial topography it does not make really sense to take the bounding box of the netherlands because we would also tile a lot of the north sea which is not really useful
so we made a table with the seed boxes bounding boxes around the netherlands in in a more efficient way so that most of the land of the netherlands is covered that way and the the pre-tiling of topography and other layers is is a lot faster well this is not
readable also but there is a configuration file where you can really adapt the number of web caches the connection to the to the database passwords for for tomcat etc etc
and i'm really going fast this was it already okay thank you any questions
i did i did and that was but some time ago and the commands were some thing like for example the png 24-bit to 8-bit conversion they didn't like the way we did it and for
other items they didn't take it up because they found that the classes we used we adapted into your web cache would not be their choice of of using our adaptations so they didn't take our work or less no i'm not sure that the the talk will be distributed sometime or
what we're asking all the presenters to do is upload it to elo geo yes i will do so because
you have 10 seconds to write down this link okay i won't i won't show it i won't do it i'll i'll upload it and okay we've got plenty of time yes but i don't i really had to go i really had to go i have to catch your flight sorry the main the main link is www.idgs.nl