Managing update of tiles of dynamic data
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 95 | |
| Author | ||
| License | CC Attribution - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/15552 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Place | Nottingham |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSS4G Nottingham 201352 / 95
17
25
29
31
32
34
48
50
56
58
68
69
70
82
89
91
00:00
AerodynamicsBoom (sailing)World Wide Web ConsortiumLevel (video gaming)Zoom lensMereologyFluid staticsSymbolic dynamicsTask (computing)Interface (computing)Line (geometry)Configuration spaceTemplate (C++)NumberAreaZoom lensBasis <Mathematik>View (database)Multiplication signTesselationPublic domainDatabaseCountingData managementPhysical systemService (economics)Total S.A.Web pagePlanningLevel (video gaming)Moment (mathematics)InformationComputer programStress (mechanics)WhiteboardExtension (kinesiology)Centralizer and normalizerTask (computing)Workstation <Musikinstrument>Client (computing)Projective planeFreewareNetwork topologyCASE <Informatik>Instance (computer science)Natural numberAddress spaceCollaborationismArithmetic meanComputer hardwareDynamical systemOffice suiteConfiguration spaceRepresentational state transferTessellationArithmetic progressionCache (computing)Web 2.0Computer fileDot productGoodness of fitEstimatorWebsiteMereologySoftwareSoftware developerServer (computing)GeometryQuery languageDifferent (Kate Ryan album)Template (C++)Computer animation
09:13
Cache (computing)BitComputer architectureSoftwareUniform resource nameOvalDatabaseMereologyConfiguration spaceEmulationMUDTessellationDegree (graph theory)Link (knot theory)MultiplicationMultiplication signType theoryMereologyTable (information)Cycle (graph theory)Data conversionLine (geometry)InformationElectronic mailing listLevel (video gaming)Data managementArithmetic meanINTEGRALTesselationObject (grammar)PlanningObservational studyNumberWeb pageHuman migrationWhiteboardDatabaseInstance (computer science)Computer programBarrelled spaceAdditionBitRow (database)TelecommunicationTask (computing)Computer configurationOrder (biology)Service (economics)MathematicsInsertion lossTemplate (C++)Zoom lensWebsiteView (database)Process (computing)Flow separationData transmissionMoment (mathematics)Incidence algebra2 (number)Link (knot theory)System callCuboidWeb 2.0Numbering schemeTessellationSocial classConfiguration spaceCache (computing)InternetworkingTransformation (genetics)Proper mapComputer architectureSoftwarePresentation of a groupGeometryAdaptive behaviorArithmetic progressionPerspective (visual)Computer fileCategory of beingServer (computing)Office suiteConnected spacePasswordComputer animation
18:27
Presentation of a groupTessellationDegree (graph theory)Link (knot theory)Multiplication signLink (knot theory)Computer animation
Transcript: English(auto-generated)
00:03
I'll try to make it, keep it brief. Good afternoon everybody. Just after 12. My name is Rob Stoecklenburg. I'm working for E-DAISH in the Netherlands, a software developer.
00:20
E-DAISH is a small company in the Netherlands, a geo company that has clients among mainly the governmental organizations, provinces, councils, police and we were once involved in a project, a national project where we had to use tiling but where the map data was updated very regularly
00:42
and we had to meet and solve some challenges in that respect and develop special software to do what we wanted and to keep up with the goals of the program. The, oh I'm going much too fast, sorry. Okay. The Dutch government has a national facility
01:06
for showing spatial plans at all levels, council, governmental, provincial, all kinds of spatial plans that are useful for public use and professional use and at the moment we have
01:23
around 15,000 plans in the system and it's growing daily. It's really taking off. It's publicly available through a website and besides the website there are facilities for professional use as well. You see every, we have 15,000 plans, every dot is a plan and it's almost the
01:47
view of the Netherlands is coming out through all the small dots. Users have to be able to zoom in to plans, to pan and to zoom out. So we need a good performance and that was one of the
02:07
demands that the project had. I show you an example, a plan more in detail from the hometown of our company Raisin in the east of the Netherlands and the website gives wonderful
02:28
means of searching. I just found this plan in our town by typing in the address of our company's headquarters and it's a small plan that says something about how to develop the
02:46
center of this town. All the documents that belong to it, all the official documents can be found as and this is a really common local plan so to say. It's difficult to translate the specific items
03:04
into English but you can say this is a local plan and each local plan has about four to five map layers including borders, including an overview layer and these five layers are
03:20
used for the majority of plans in this facility and they all constitute the whole service area of the Netherlands. Maybe a few percent of the total number of plans is different, they have often a larger service area, they are more in the national area or the provincial areas of
03:47
special plans and they have their own specific layers with them and that makes one of the challenges in this case. The topography, the map of the town and surroundings that you see
04:04
is pre-tiled because it changes maybe once a year or even less so we do that on separate hardware. In this spatial plans project plans are updated daily because councils and everybody
04:24
who is using it from the government really wants to be able to present the new data as soon as it is ready so we have from a few plans a day to a few hundred plans a day per day that are updated in the system and have to be shown on the map as soon as possible. We provide
04:46
for a number of zoom levels from the overview of the Netherlands up to the utmost detail of a certain plan and we need to do that for different map layers as well. Most plans contain several layers like the plan I just showed before it contains five layers
05:05
and some of these special plans contain 10 or more layers as well and we have to provide for in total we have 700 map layers in the system that we have to provide to users.
05:24
To give you an example of how many tiles that could be for the topography layer I have been counting the number of tiles for the whole of the Netherlands for all these zoom levels and that's for one layer only five and a half million tiles and we have 700 layers in total
05:47
and this means that if we would not use tiling it would result in high demand on the WMS because at the daytime the map server is also used by professionals as well and
06:02
it serves the map data for the website. So we had to come with the solution and we had to use tiling to diminish the demand on the services but what we only wanted to do is tile
06:20
parts of the layer that were changed. We can't change the whole layers every time and the customer also had said that they wanted to be ready with tiling before office hours so there would not be any stress on the WMS during office hours.
06:43
We came up with a solution that we call the tiling manager. Static data is pre-tiled like topography and dynamic data which can change per day is tiled before office hours. We use the web cache which is a perfectly sound solution for for tiling for static tiling that is
07:04
so in this case we're not smiling yet we do need some something extra and that is the tiling manager which sits between it sits between the planned database of the whole project and the web cache and it queries the planned database on a daily basis
07:25
and then makes tasks that it can issue to geoweb cache and we had to change to adapt geoweb cache as well to be able to communicate it with it in in a way that we wanted and to to be able to monitor the progress of geoweb cache. We faced a number of challenges as I said the
07:53
monitoring of geoweb cache was wanted and we also wanted to be able to to stop geoweb cache because the tiling manager can get a command to stop everything it is doing at that moment so it has
08:05
to stop the geoweb cache instance as well with the tasks at hand and we had to extend the rest interface of geoweb cache. The tiling manager itself has a simple information page
08:21
which shows the plans at hand all the layers per plan the zoom level to which the layers have to be tiled and during tiling it gives the percentage of tiles ready and also an estimate of the time still to to expect the tiling will will need to to go on and because we have several plans
08:52
that that have their own their own layers the geoweb cache has a configuration file with the all the map layers inside and we only have five of these common layers that are common for a
09:04
lot of plans but there are some plans that use specific layers which we can't predict beforehand so we have to generate a configuration for geoweb cache using a template and the tiling manager knows about the plans it knows the template it fills in all the details
09:22
and it makes a proper configuration for geoweb cache that it needs we also did some work on optimizing the wms performance mainly by by using some extra caching so the the database requests were diminished the number of them and we
09:49
also found it very handy to be able to shrink the pngs that were produced 24-bit pngs to 8-bit using png quant which is a a standalone program that we use for every child
10:05
and the transformation is also built into geoweb cache and that means that the the i think the size of these tiles is indeed going from 100 to 50 or 30 percent size so it's really helpful to get less transport over the over the internet
10:31
and what was that was that we had up-to-date tiles so when a council changes a plan today and they upload it to the site they really want the map and the view in the next day
10:47
to be in order we can't have old tiles running around or or still being there so up-to-date tiles were was very important and this was ensured by the cycle that we made
11:02
we built into this tiling manager we have the plan database and the tiling manager takes all the plans that were updated deleted or or added it finds all the layers in all these plans it makes
11:21
for a task for each layer a geoweb cache task and all these tasks are visible in the information page of the tiling manager and the cycle starts with generating a new configuration so for all the plans that we have in in the morning so to say we generate a new configuration for geoweb cache
11:42
up to date then we're going to truncate to delete all the tiles of the changed plans of the of the of the the day before so we erase the old situation delete all the tiles that that were changed
12:02
next we send the new configuration that we generated here we send it to geoweb cache and we reset geoweb cache because we have to make sure that geoweb cache can read this new configuration file this was the most safe option we had then after that and
12:27
geoweb cache is back online we give it seat tasks so it makes the tiles based on the new configuration and it makes tiles which represent the new plans and in the map later on
12:42
and all these tasks in this info page are being done bit by bit until the list is empty and then we're ready for the day and hopefully before office hours a small idea
13:03
about the architecture there is one tiling manager and there can be multiple geoweb cache instances and they can each run in they run in a tomcat and they can run all in the same tomcat
13:23
or in a separate tomcat or on separate servers communication is via http so there is a flexible way that you can deploy the tiling manager and the geoweb cache instances you can have many geoweb cache instances if you want but they all do get the same configuration
13:43
and the same configuration means that they all send their requests to the same wms to get and produce the tiles the tiling manager is trying to find when it goes through its list through the cycle and through the list of tasks it finds a geoweb cache that is
14:05
doing nothing at the moment and send a task to it so it can do the tiles for that plan for that layer and then finds the next geoweb cache that is doing nothing and send another layer to that one and so forth and if they're all busy it just waits and monitors progress and waits until one
14:24
of these geoweb caches is free or the list of tasks is is empty and all that is configured in a properties file for the tiling manager it's a bit bad well it's not readable really but
14:45
from a software perspective there is a tiling manager class that accepts http requests from users translates that into calls to a manager service and the manager service has a pool of
15:00
that holds w no geoweb cache instances in the pool we have several geoweb caches representing geoweb caches and and the status of geoweb caches is also monitored in in a special class in a special object all is written in in java
15:21
and i think that is that if i show a very short overview of the database scheme there is a table that that contains the the task the cycle of of the day so to say a row for the cycle
15:44
of the day there is a big table that contains all the status items that are important for the for the task at hand the the plan the layer type the the zoom level we need to go to answer etc etc then there is a table that combines the type of plan with the layers that the plan
16:05
is normally normally containing and we have for the pre-tiling for for example when we want to pre-trial topography it does not make really sense to take the bounding box of the netherlands because we would also tile a lot of the north sea which is not really useful
16:24
so we made a table with the seed boxes bounding boxes around the netherlands in in a more efficient way so that most of the land of the netherlands is covered that way and the the pre-tiling of topography and other layers is is a lot faster well this is not
16:46
readable also but there is a configuration file where you can really adapt the number of web caches the connection to the to the database passwords for for tomcat etc etc
17:06
and i'm really going fast this was it already okay thank you any questions
17:24
i did i did and that was but some time ago and the commands were some thing like for example the png 24-bit to 8-bit conversion they didn't like the way we did it and for
17:41
other items they didn't take it up because they found that the classes we used we adapted into your web cache would not be their choice of of using our adaptations so they didn't take our work or less no i'm not sure that the the talk will be distributed sometime or
18:15
what we're asking all the presenters to do is upload it to elo geo yes i will do so because
18:21
you have 10 seconds to write down this link okay i won't i won't show it i won't do it i'll i'll upload it and okay we've got plenty of time yes but i don't i really had to go i really had to go i have to catch your flight sorry the main the main link is www.idgs.nl