We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Design and development of the UN Vector Tile Toolkit

00:00
subtitles
off
playback rate
1
subtitles
off
English (auto-generated)
playback rate
0.25
0.5
0.75
1
1.25
1.5
1.75
2
quality
576p
22
 Cite
 Share
 Related Material
 Download Purchase
No logo availableFOSS4G
Logo of Open Source Geospatial Foundation (OSGeo)Open Source Geospatial Foundation (OSGeo)
 Hayashi, Hirofumi Fujimura, Hidenori Martin, Oliva SATO, Takenori IWASAKI, Nobusuke Ferreiro, Diego Gonzalez Kayama, Yoichi Motojima, Yusuke Mugambi, Francis

Formal Metadata

Title
Design and development of the UN Vector Tile Toolkit
Title of Series
Number of Parts
295
Author
Contributors
License
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
The UN Open GIS Initiative is catalyzing the development of an open community for basemap vector tile providers. The Initiative promotes the UN Vector Tile Toolkit, a package of open source tools designed to enable public basemap providers, such as the United Nations geospatial information services or governmental mapping organizations, among others, to deliver their basemap vector tiles leveraging the latest web map technologies. The toolkit provides a set of Node.js open source scripts designed for developers to use in conjunction with existing and proven open-source vector tile software (such as [Tippecanoe](https://github.com/mapbox/tippecanoe), [Maputnik](https://github.com/maputnik/editor), [Mapbox GL JS](https://github.com/mapbox/mapbox-gl-js) and [vt-optimizer](https://github.com/ibesora/vt-optimizer)). The toolkit will help organizations to produce, host, style, and optimize fast and interoperable basemap vector tiles, making them available with various application frameworks. This paper presents the methodology followed to support the requirements from the UN Peacekeeping operations’ point of view. The significant challenges were to ensure (1) flexibility of the software so that it works with different source data, including UN mission-specific basemap data and global OpenStreetMap data, (2) automatic and continuous update of global basemap vector tiles, (3) fast vector tiles that can be used with mobile devices in the field, and (4) interoperability with existing enterprise geospatial software frameworks. We approached these challenges by introducing "on-the-fly vector tile schema modification" to the Toolkit so that an agile adaptation, improvement, and optimization of the vector tile schema is possible. This method also reduces the use of temporary storage and improves the processing time by exploiting the outstanding stream processing capabilities of Node.js and Tippecanoe. We also adopted an approach to produce basemap vector tiles in modules, so that the processes to produce and deploy the vector tiles are well pipelined. This modularization approach also extended the scalability of the Toolkit by allowing parallel or even distributed production. Quantitative measurements, like in vt-optimizer, were other dominant methodology adopted in the Toolkit that enabled evidence-based performance tuning and production time reduction the vector tiles. As a result, we successfully implemented automatic and continuous updates of basemap vector tiles from a continuously updated PostGIS database. The paper also covers how the project ensured interoperability with different existing enterprise geospatial software frameworks that use less-advanced web map libraries. The project aims to build a sustainable community of developers supporting the provision of fast and interoperable basemap vector tiles that meet the requirements of various users in the field missions and UN headquarters among others.
Keywords
00:00
Array data structurePresentation of a groupInformationMachine visionOpen sourceStreaming mediaGame theoryPrisoner's dilemmaVector spaceTesselationMachine visionSelf-organizationPresentation of a groupComputer programmingSystem callMappingComputer animation
01:09
Machine visionArray data structureOpen sourceGame theoryStreaming mediaOpen sourceSelf-organizationWeb 2.0Texture mappingForcing (mathematics)Video gameGame theoryMachine visionCartesian coordinate systemIntegrated development environmentDecision theoryComputer animation
01:57
Vector spaceProduct (business)Array data structureSoftware developerComputer networkOperator (mathematics)SequenceMathematical optimizationExpressionServer (computing)Computer-generated imagerySoftware frameworkExecution unitAdditionSelf-organizationTexture mappingComputer programmingNeuroinformatikOpen sourceSoftwareFile formatEntropie <Informationstheorie>Observational study10 (number)Streaming mediaMathematical optimizationInternetworkingBitCASE <Informatik>Vector spaceTessellationTesselationComputer simulationProduct (business)SequenceOperator (mathematics)Different (Kate Ryan album)Communications protocolProcess (computing)Computer animation
04:49
Vector spaceArray data structureSoftware frameworkServer (computing)Computer-generated imageryBitCartesian coordinate systemMultiplication signFile formatVector spaceTessellationTesselationSingle-precision floating-point formatAreaOpen setComputer animationMeeting/Interview
05:20
Array data structureVector spaceComputer-generated imageryServer (computing)Software frameworkConcurrency (computer science)Task (computing)BefehlsprozessorDivision (mathematics)Decision theoryStreaming mediaModul <Datentyp>Process (computing)Open setTexture mappingMeasurementVector spaceLibrary (computing)TessellationProduct (business)Cartesian coordinate systemDivisorNeuroinformatikStreaming mediaMaizeReading (process)Projective planeFocus (optics)Electric generatorObservational studyComputer animation
06:42
BefehlsprozessorDecision theoryStreaming mediaModul <Datentyp>Process (computing)Visual systemView (database)Reduction of orderTask (computing)Meta elementModule (mathematics)NumberConcurrency (computer science)CASE <Informatik>NeuroinformatikStreaming mediaTask (computing)Operator (mathematics)Meta elementInstance (computer science)SoftwareProcess (computing)Level (video gaming)Content (media)TessellationVector spaceOcean currentBitAreaElectric generatorSet (mathematics)TesselationTwitterOpen sourceDirected graphLecture/Conference
10:04
Array data structureMassVector spaceApproximationComputer animation
10:37
Compact spaceServer (computing)Interface (computing)Enterprise architectureIntegrated development environmentVector spaceOpen sourceSoftware frameworkArray data structureComputer programmingObservational studyOpen sourceProjective planeQuicksortSoftwareShared memoryActuaryInterface (computing)Vector spaceTessellationTesselationComputer animation
11:27
Array data structureVector spaceExecutive information systemGrass (card game)Vector spaceTesselationTexture mappingClassical physicsTessellationService (economics)Latent heatObservational studyMultiplication signComputer animation
12:03
Vector spaceArray data structureMassComputer-generated imageryDemo (music)Interface (computing)Compact spaceServer (computing)Open sourceSoftware frameworkEnterprise architectureIntegrated development environmentMIDIOperator (mathematics)BitSoftwareCurveComputer animation
12:28
Computer-generated imageryArray data structureDemo (music)Execution unitBroadcast programmingOpen setTexture mappingStaff (military)Computer networkPresentation of a groupRevision controlDISMAInformationData managementLie groupDemo (music)Heat transferCompilerIntegrated development environmentEvent horizonGroup actionExecution unitForcing (mathematics)MappingComputer animation
13:08
Texture mappingBroadcast programmingOpen setStaff (military)Presentation of a groupComputer networkArray data structureRevision controlDISMAInformationData managementMappingSelf-organizationGroup actionLecture/ConferenceComputer animation
13:43
File formatReading (process)MathematicsKey (cryptography)Game controllerBoom (sailing)Operator (mathematics)Open source40 (number)Boundary value problemShared memorySystem callMaxima and minimaMobile appData qualityPhysical systemOrder (biology)State of matterAreaQuicksortWebsiteMoment (mathematics)TwitterMetropolitan area networkSoftwareProcess (computing)Open setProduct (business)Texture mappingProjective planeCASE <Informatik>MereologyPopulation densityHand fanLatent heatBuildingGroup actionDirected graphTessellationMappingExpert systemGoodness of fitReal numberLevel (video gaming)Vector spaceMechanism designCartesian coordinate systemComputer fileProcedural programmingShape (magazine)Zoom lensTesselationBitLecture/ConferenceMeeting/Interview
Transcript: English(auto-generated)
00:07
Thank you very much for your attention. Our last presenter is Hidenori from GSI Japan. He will introduce design and development of UN vector tile toolkit. Please welcome him with big hands.
00:25
Okay. Yes. Thank you, Hyeyoung. So I would like to talk about UN vector tile toolkit. Yes, my name is Hidenori Fujimura and I'm working in geospatial information authority of Japan now.
00:40
Regarding the co-authors, actually, I'm not alone. We are actually ten co-authors. So regarding the co-authors, please refer to our papers. Yes, we are actually from United Nations, National Mapping Agency and National Research Institute and also private companies. We are joining together to solve the problem of vector tile toolkit in public organizations.
01:05
Okay. Yes, so at first I would like to share our vision. Yes, the thing with this. Yes, we are thinking about what if the public organization base maps are responsible like in video games. Then, yes, we have more
01:23
active application or if there is some limited environment, we will have some opportunity to make the offline map. So such kind of thing is what we are seeking. And yes, maybe, yes, I would like to say public organization
01:42
also deserves the best web map technology. And it's open source is a kind of method to unite the forces of public organizations because otherwise it is a little bit difficult for public organization to join forces. So yes, our product is actually twofold. One thing is the software itself called UN vector tile toolkit.
02:05
This is to support especially public organization to produce or host or optimize vector tiles and style vector tiles. And another big product is actually the network of developers and operators
02:21
because yes, we can share the problems and we can share the actual solutions. So at first I would like to talk about the software itself. But actually the software itself is not so very new because we are using existing open source software.
02:40
We have already several very good software to produce vector tiles. But the problem is it is a little bit too hard for public organization to make use of these existing software. So actually we are using this kind of existing software. Maybe, yeah. Typicano or Maputonic or yes, vector tile optimizer developed by Isaac
03:05
from Spain. So we are using this kind of very excellent software. The problem is how we can make use of this software inside the public organizations. Yes, in public organization there is actually many
03:24
different format, different data. So the challenge is how we can work with diverse data and large data. So we need to work with gigabytes or tens of gigabytes of data to produce our base map.
03:41
Yes, this is more technology-oriented things, but we try to stick on to stream-based computing model. By using the stream-based computing model, we can easily mix or fix or handle the streams concurrently so that we have more freedom to optimize the
04:05
processing to produce the vector tiles. And we stick to use the GeoJSON text sequence format as a kind of wire format to connect these existing software. And yes, this is a major achievement from our software. So the biggest thing is that we could produce
04:26
vector tiles of the globe from UN internet post GIS base map data and using only single map, sorry, MacBook Pro. And yes, maybe I need to explain what is a UN base map. UN base map is, yes, in this case
04:44
90% is actually the OSM data and other 10% is random-use data or other POI data. So I would say maybe this is a little bit more from OSM data. So it's about 50 gigabytes coupled in the globe if we use the PBA format.
05:02
And the other thing is the interoperability. If we use single vector tiles, if the vector tile is used by only one single application, it is not so useful, but we think that single vector tiles should be used by different
05:22
applications. So we developed some interoperability layer so that we can use these vector tiles from open layers or other commercial libraries. But today, yes, I would like to focus on this, the first thing,
05:42
production of vector tiles in these 80 hours. Why 80 hours? The speed is important because we need continuous update. We are managing the base map data and the base map data are updated daily. So we need to
06:03
make the automatic and continuous update of our base map. And in the public organization, it is not so easy to get plenty of computing infrastructure. So we need to reduce the amount of computing infrastructure we need to use. So efficiency is very important for our project.
06:26
And yes, I would like to share three measures I take to make the 80 hours update possible. First one is very classic one. We divide and rule the data set. The
06:43
main data set is around 50 gigabytes, but it's handling 100 streams of 1 gigabyte is very easy for by using the computer of current generation. But yes, single 100 gigabyte stream or
07:01
10,000, 10 megabyte stream is a little bit harder to handle. So we choose the Zumu level 6 module as a basic module to produce the vector tiles. And so that we can produce these modules in parallel or in concurrent manner
07:21
so that we can make use of the resource for IO or CPU efficiently. So if one task is working on maybe exporting, importing, other things, other task is working on creating vector tiles, computing. So we can use both IO and the CPU concurrently.
07:45
And yes, another thing is also a very heuristic one. So if we see the earth, 70% of the earth is actually ocean and we choose that this module is maybe hundreds of kilometer per hundredth of kilometer, but we actually checked the data content itself and
08:06
actually more than 1600 modules are actually empty in OSM data. So we just simply skipped these 40% of the modules so that we can concentrate on the
08:21
60% of the modules. So and this is the last one. So when we create the vector tile for each modules, we extract data for these modules from the planet OSM data, but scanning planet OSM data is very expensive one.
08:42
So we make a kind of meta module to make the extraction process fewer. So we choose to create 12 modules and actually in the case of OSM data,
09:02
OSM data is about 50 gigabyte, but this small area number four and number five it takes about 10 gigabytes. So world is not so equal in the case of OSM data. So we carefully choose these meta module, which is around 10 gigabyte so that we can reduce the number of scanning the planet OSM data. So this makes
09:29
this possible. So we created vector tiles covering all the globe within 79 hour and 22 minutes. So yes, I was successful to make it possible and
09:46
yes, eight hours in one PC with one PC means maybe we have 10 instances then it will take just eight hours. So the daily update will be possible, I think.
10:00
And yes, I'd like to talk about a network of developers and operators. Actually, yes as of now, we are a team of and it's OSOJP and it's National Research Institute for Agriculture and yes
10:22
from August, one engineer from Mapbox joined this voluntary initiative. So yes, we are actually very diverse team to create this effort and we are sticking to tackling common problems because we would like to make it open source project
10:45
so we carefully choose very common problems like this and also we are facing these new sorry, problems we need to take care of. So regarding the first two things, yes, there is some software to make vector tiles, but
11:07
we need to share the actual know-how to produce the actual vector tiles and another thing is the interoperability. Someone says vector tile is useful, but maybe it will not work with my software. So this kind of
11:23
fear we have to solve this. And yes, United Nations Secretariat and the GSI, National Mapping Obezese of Japan shares a lot of knowledge and actually, yes we have produced that vector tile of United Nations Secretariat, but yes from National Mapping Obezese of Japan
11:43
we started vector tiles from the end of July. So yes, the classic topography map is now in Mapbox vector tile specification and we are doing some kind of experiment with these new services and this knowledge will be shared within
12:03
these teams and so that we can contribute to the better UN operations. So this is a way we can go. And yes, another thing is that it is still too hard to share how to
12:20
use this software. So I mean we need to have less steep learning curve and the most difficult thing is how we can prepare the infrastructure or OS or compiler or this kind of thing. So we introduced some Docker environment so that we can start from the
12:43
working environment and it will work with even with Raspberry Pi so that we can have more opportunity for technology transfer or demo or maybe hands-on. And yes, this hands-on material will be ready in the middle of September in 4G Niigata.
13:02
Yes, we are trying to expand this initiative by making use of some events we have. So as a mapping agency, we have some opportunity for UN conferences like UN-GG-IM and we have some Asian Pacific chapter or a working group on disasters. So making use of this opportunity
13:24
we would like to share this kind of new technology for the use in public organizations. Okay, thank you so much.
13:45
So, thank you. It's very interesting. I think also useful. My question is because you didn't speak about the license. So which is the license you will make available this software? Yes, yeah.
14:02
As far as I know for example typical new or this kind of existing open source software is of course open source license like MIT or something and what we I have developed, what we have developed is now under UN license. That is a little bit less than the open source license.
14:22
But maybe if it is necessary to work on the OS Azure incubation process maybe we need to kind of upgrade to MIT license or something. In mapping agency we have some experience to work with PSD license or such kind of license. So we will be ready to provide a proper license.
14:43
Thank you. Okay, first. How do you control? Okay, how do you control the data quality coming from OSM?
15:01
Because one thing is like yeah, you can easily make titles, but yeah, there's also a lot of areas. Yes. Yes. Yes, it's not mapped. And how do you make sure that the users actually know that? Okay. All these buildings are not mapped. Okay. Thank you so much. So yes, we are working as a kind of open source initiative called UN
15:25
OpenJS initiative and there is other working group working on the data quality issues and actually, my project is focusing on the production of the vector tiles data. So we are
15:40
concentrating on the process. So maybe the answer is that things must be processed outside this project. Yeah, I'm chief of the UN GIS and we are working very closely all together.
16:03
But there is another initiative as one maybe initiative and also we have our GIS Center in Italy as a big you know Center. We have a lot of good experts. So they are continuously improving and then up and also contributing to the OSM data as well as that the real operation data and so on and so forth.
16:22
So we like to build up more good mechanism. How we can actually build a more good quality of the data in general to be used by the public consumption. And if you are interested on that activity, we call it One Map. So there
16:41
Diego Gonzalez from the UN GIS is here. Yeah, so you can call him and talk much more. Is there something, you know, Diego you'd like to mention? It looks like all of us are working for the UN almost.
17:06
So in Brindisi, we are also working on the data part. We are actually creating what we call now the UN street map which is really the equivalent to open a street map plus what he already mentioned with this 10% of
17:25
UN only UN data. And the reason is because we cannot use obviously open a street map as such for a UN mapping applications. We cannot rely on open a street map for
17:42
spelling of towns or villages or the boundaries of international countries. So that has to be kept within the UN and that's why we are trying to focus on the quality of the data specifically and at this moment in the areas of operations of the peacekeeping
18:04
missions. So mainly in Africa, Middle East, some parts of Latin America. So that's what that's where we are focusing on on a specific data quality. We are also contributing to open open a street map and we are adding or we are trying to add quality data and
18:27
and also the density of the data of over those areas. And if I may, I would like to add some example from the mapping agency of Japan. So in the case of national mapping agency of Japan, we think the vector tile as a kind of new format for dissemination.
18:44
So we provide our data, our quality control data as a shape file as of now, but we need to add some additional format. That is the vector tiles. So we already have some procedure for quality control before we change to the vector tiles.
19:03
So that is the idea from the mapping agency. Thank you. Thank you. And we still have a five minute. Is it okay? Sato, what zoom level do you generate the tiles?
19:22
Excuse me. Zoom level. Zoom level. Yes. Yes. Maximum zoom level is as far as I remember is maybe 14. 14. Yes. I started from 15, but I thought 14 is enough. Is there any questions?
19:40
I see none. Okay. If you have any questions. I think that this is one joint project among our, you know, member state and UN that how we can really benefit to the public organization of the mapping agency of the countries and so on and so forth.
20:02
For those who are actually working closely with the national mapping agency, particularly for the developing countries mapping agency, please just share that this is a new technology that we are very much happy to be support and assist together with the GSI Japan. So they can actually work together
20:23
and then also building together and train them, but also you know, continuously transfer the technology to the developing country. This is one of the also important purpose. Thank you. So is there any questions to him?
20:44
See none. Thank you very much. Thank you so much. Okay.