OGC GeoPackage in practice: Implementing a new OGC specification with open-source tools

Video thumbnail (Frame 0) Video thumbnail (Frame 10723) Video thumbnail (Frame 21355) Video thumbnail (Frame 22907) Video thumbnail (Frame 24657) Video thumbnail (Frame 25511) Video thumbnail (Frame 32333) Video thumbnail (Frame 37948)
Video in TIB AV-Portal: OGC GeoPackage in practice: Implementing a new OGC specification with open-source tools

Formal Metadata

OGC GeoPackage in practice: Implementing a new OGC specification with open-source tools
Title of Series
CC Attribution - NonCommercial - ShareAlike 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
GeoPackage is a new encoding standard created by the Open Geospatial Consortium as a modern alternative to formats like SDTS and Shapefile. Using SQLite, the single-file relational database can hold raster imagery, vector features and metadata. GeoPackage is an ideal data container for mobile devices such as smartphones, IoT devices, wearables, and even automobiles. We have created a few open-source tools to manipulate this exciting technology in a way that is useful to the geospatial community. Our goal with the GeoPackage specification implementations is simple: Create GeoPackages quickly and reliably while maintaining standard conformance. The single biggest issue we have faced is the speed in which large amounts of imagery can be disseminated to the end user. Data standards reliability was also a concern because we found many vendors interpreted the specification differently or to suite their own needs. Finally, the main problem GeoPackage was created was to solve was interoperability. We set out to create an implementation that would guide other parties towards making a data product that would function as well on one platform as it would on a completely different platform. Our initial implementation of the GeoPackage specification was created using Python 2.7.x. The software design was intended for command line use only in a script-friendly environment where tiling speed was paramount. The Gdal2tiles.py script was improved upon by harnessing the Python multiprocessing library so that multiple tile jobs could run simultaneously. The other piece of the workflow, creating GeoPackages, would be a separate development effort from scratch called tiles2gpkg parallel.py. In tiles2gpkg parallel.py, we implemented multiprocessing by writing to separate SQLite databases in parallel and then merging the tiled data sets into one compact database. This implementation worked well and increased the performance of producing these data sets; however, the command line design means that all but the most technically adept users would struggle to use the tools. With the initial Python implementation getting early-adopters a preview of GeoPackage in the short term, our team set out to make a production-quality GeoPackage API that could satisfy all user needs. Named Software to Aggregate Geospatial Data or SWAGD, we created a robust library for tiling raster data, packaging raster data stores into GeoPackages, and viewing either the raw tiles OR the finished GeoPackage products within a map viewer. Additionally, a Geopackage verification tool was created to foster community adoption. For more information, see our Github site here: https://github.com/GitHubRGI/swagd. Many open-source tools are being leveraged on the SWAGD project, including many common build and continuous integration tools including Github, TravisCI, WaffleIO, and Coverity. Using proven software development mechanisms like unit testing and code reviews we now have a consistent, reproducible, and inclusive GeoPackage implementation. We have an aggressive list of future capability that we would like to develop including ad-hoc routing on a mobile device, vector tile data sets, and even 3D support.
Web page Ocean current Building Presentation of a group Implementation Server (computing) Open source Computer file Sequel State of matter Java applet Multiplication sign 1 (number) Set (mathematics) Online help Shape (magazine) Mereology Metadata Revision control Medical imaging Bit rate Core dump Reduction of order Energy level Extension (kinesiology) Error message Plug-in (computing) Information Projective plane Sound effect Bit Database Instance (computer science) Limit (category theory) Demoscene Tessellation Process (computing) Computer animation Vector space Software Personal digital assistant Chain Order (biology) Universe (mathematics) Figurate number Family Arithmetic progression Row (database)
Web page Android (robot) Server (computing) Implementation Computer file Code Multiplication sign Survival analysis Function (mathematics) Binary file Medical imaging Latent heat Component-based software engineering Single-precision floating-point format Formal verification Core dump Cuboid Energy level Software testing Data structure Extension (kinesiology) Social class Task (computing) Scripting language Information Mapping Tesselation Software developer Projective plane Data storage device Database Instance (computer science) Evolute Cartesian coordinate system Demoscene Tessellation Cache (computing) Uniform resource locator Pointer (computer programming) Process (computing) Computer animation Software Query language Order (biology) Self-organization Right angle Window
Greatest element 1 (number) Mereology Metadata Medical imaging Mathematics Different (Kate Ryan album) Formal verification Cuboid Software testing Physical system Task (computing) Information Tesselation Sound effect Multilateration Instance (computer science) Cartesian coordinate system Tessellation Process (computing) Computer animation Personal digital assistant output Speech synthesis Right angle Spacetime
Axiom of choice Android (robot) Building Implementation Functional (mathematics) Sequel Link (knot theory) Open source Java applet Code Multiplication sign Raster graphics Mereology Binary file Dimensional analysis Number Medical imaging Inference Different (Kate Ryan album) Operator (mathematics) Core dump Software framework Software testing Metropolitan area network Physical system Area Email Forcing (mathematics) Software developer Projective plane Database Instance (computer science) Line (geometry) Lattice (order) Limit (category theory) Tessellation Process (computing) Computer animation Personal digital assistant Buffer solution Universe (mathematics) Website Right angle Pressure Window
Point (geometry) Implementation Presentation of a group Overhead (computing) Computer file Sequel Multiplication sign ACID Similarity (geometry) Online help Mereology Medical imaging Goodness of fit Roundness (object) Lecture/Conference Different (Kate Ryan album) Data structure Extension (kinesiology) Data compression Task (computing) Pairwise comparison Texture mapping Information Binary code Content (media) Calculus Instance (computer science) Tessellation Personal digital assistant Blog Right angle Table (information)
correct about 11 25 now so of it started and every once the winter and they suffer engineer at r G I will talk to you about implement OGC GeoPackage would you had been a new standard that OGC's and recently and this is for containing featureset data and metadata metadata schemas on the collected this so a little bit more about me and from out their backs Virginia in the United States by an experience and caching storing large rested images so the work I've been doing for about the past 3 years has been involved with an with using do not quite often to take these large after images used during his visit the images and and cutting them into many pieces and and there are a lot of methods for doing that the main benefits being that's and it greatly increases by the time it takes to actually render that information so I and then the work we been doing for the past 3 is has been implementing that men encoding standard so we initially started up with Python and that progress into Java and then from job we we also want to an interim and this with some others as well JavaScript but has gone you got it so are actually about projects we will know more about suffered Agassi aggregated geospatial data but was this like the and GeoPackage Python which was the 1st naive implementation we started off with not so for for so ideally we were starting off with 1 8 which was kind of aggressive and if I was you to get published or made with . 1 7 a double 6 that is because I'm not a lot of people all are comfortable going to the latest version of Geller typically you'll find a lot of people with 1 6 0 1 7 of the wild and there can reluctant to go to go any further news that way you know there's a lot of a of benefits for going to just 1 it we saw that we decided died in infancy so but you see smaller presentations we have going on the yesterday and church going to be on the website made in France had talked about due packaging and habits change the way domestic standards and also tomorrow I'll be presenting some of the open-source build tools that I have incorporated into and my projects in order to help me and help my team and you know reduce risk producer errors the so so what is the d of Act including and Rossiter talking about and the actual implementation we made there are many in our approach to implementing encoding standards and and what we learned building the reference imitation and the lessons learned from from this experience there numerous the so Due baggage including standard is a set of conventions for storing following in this is all available on the website so that a mention of family to accept which is that georeferenced image current many pieces the and and we that scale so that they basically have a pyramid dataset and during collected this also steam attached to their and describing utile set describing what levels it exists that how many there are and a lot of other of other helpful things that only get then the tiles immortelles kind of dumps a lot of tiles into a single sequel a database of details like the frozen rigor around that so you have more information woman and the at the university you you can describe your that ever did you have more maybe you wanna talk about classification or or because there a limit and then extensions so they tend to those that I think is 1 more powerful features of an extensible either do think that the current standard which right now I will not let you so things like that perhaps you add a network dataset into your D effect perhaps you want to add timestamping due to your images to files because tells can update has been changing you wanna figure out which ones are 1 the freshest or or the most stable for instance so when we started um the this is something at the bottom actually had to give a piece of land of presentation so how happy this is why we have another senator to make use of I think it's a little bit better than the situation here is is saying because I I saw that and tiles was was pretty good and tells a really good way to to take this idea of the cash mad and thrown onto a mobile device or or or any device as a single file that was a lot of a lot of value there so yeah but also you have to take the entire DG about have to put records in there you don't want to get the article grasses in there you don't want you can make the white with it if you if you really desire to but of I think that the the there's nothing wrong with more standard but the yeah it's going really depend on the use case so look at existing technology is already out there that perhaps could compete with you packeted something similar to what you does and the main 1 you'll see on on the GC side is the shape of so she file you'll you'll have your vector in an information item meta-data the shapefiles not gonna do your acetyl datasets and k and l who would kind of the valuable that better than state value actually on the tarmac datasets and thinking and you and I also vectors well but so you don't you don't get the nice the nice parts of the scene and metadata extensions with L today on very fast very good predictor of the tiles you could probably broke that information into and details the problem there really and as a state in the telephone supports where Mikhail where's your baggage to support any any just each other anything specific even somebody gave me your dedicated to be so honored so there are more implementers of GeoPackage and X a growing and then that than just hours g had quite a while ago for just features in in 111 0 where she's 111 1 in our hearts were now the thereafter tiles had had just emerged to 0 but we have looked at that I would personally but as you say just use right talk to some of the Jews server and the people that are here there's a community-supported plug-ins and that somebody's rate of but well and nothing nothing actually official yet as far as I know he's working from so spatial like pages and the version that that like for a month ago and it was just pretty recent but every and
has had feature support I would say and then there's the support you but it is actually still ongoing and Russia supports is coming however the know the way they deal with with tiled image tiles entailed in caches is actually fundamentally different from from T a messy that meet you deviance yes they are detection of a binary format Turkish that of situation that the negative uh really quick really quick and then digital below and they have the ability currently to to push out the imagery that you see to a GeoPackage you can't export that an effort to those because that was that was go that we reaction make about and that's the survival of naive implementation and so if you if you have a digital without you can go there and instead of getting a a file folder structure full tiles you can actually just get a GeoPackage have a simple database here it here you were supported and you can see that so our approach so all what I really want to do is create an API for for other developers and for other implementers that they could just pick and choose what they wanted so I knew that at the intended user that we were written the software for wasn't gonna need everything in a single project right you would have a certain individuals that made would want to deploy this on a server and they would want to that perhaps had users can in and they would have a tile back and on the server that they would be able to draw from take those tiles on take that imagery Tyler bear even and and push it up to a user that they could have a downloadable file or maybe they want you and on mobile and Thailand's doesn't make sense and I want to build the URI and all-round API as as just the demonstration so that you could see the testing delighted you if you if you use on a desktop users on the server or for use it on a mobile device or or whatever then we start up a double 1 8th of an then which is really great everybody on the team what's it but and there there are some problem there are some features that especially for Android and we'll talk about that later just our ongoing work in and we had backports things but and the thalamus so the core API we have a common libraries as you did your typical geospatial things you have a data store library which is going have a of the light to manipulate the store of tiles historic task to be a full file folder structure of of organize images or it could be a generic or community ties whatever you habits storing tile information cash map information that's the and that libraries would allow you to interact with us and Jia Jia ejido tertiles project as well is there exists in there and and you know the name taken from the the Python scripts by the same name and a lot of the a lot of the code in that project closely follows what they're doing and then it was just taking a in support image in time for you and and and giving you output would you buy class is is typically what what people are going to want that's 1 polite to verify GeoPackage that's to open 1 to create 1 to put the imagery into 1 and then our network extension is the is is actually an an extension that we talked about where you are able to add a network dataset rather bad for instance nodes and edges to a GeoPackage interacting query that information on your actual device like medium over location order a lot using things like a star or dextrose running the I you to set the keynote about about software interoperability is definitely something that the have to worry about costly solve wanted to have a the a lot have that kind of you know thought about before coming in and making everything so we had a verification component there that when you open the GeoPackage it's want to check literally everything all to other pages about your baggage encoding document it's going to go through and then see that if you if you've been adhering to that and and and that was so you so so popular that reaction it that's the application so that other and other people in the while other implementers and other companies could could take that and say hello make my GeoPackage interoperable do you have anything that could help ensure lecture this is this verification of of will help you I know that should give you pointers to the actual specification headlights so you can save OK I'm I'm still not following here and we would like to what is seen in the and other way we we show for interoperability sport was you know just using the JVM and you know like approach Atigun Python Haskell I could go crazy and but you know the reality is that the GTM has allowed minimize development team you use Windows Linux at 1st use Linux to do and the development and there's lots to push to Android so that was very very important that we have something that was be easily portable and the outputs that it supports are and if you never made a leaflet based network level you know you those of us the big 3 that you'll have support for baked 3 of 7 more brocaded people do that and you can do it and then global Mercator which we typically don't see a lot of people political Mercator but I think it over here only in not egregious in in the the 4 of the of the distortion that that it introduces it so we made a referent your that I mentioned and it's it's swing base which really is that the only thing it all of evolution was of an NBC and end situation of but would you know we're getting there just infectious cannot for 1 8 so that's something that's very exciting for the scene J. that you're is a job that I that happened find use in research and in that US supported tiles that sets out of the box would have to rewrite anything the problem is it was the only 1 that was out there so there are some things that it does poorly that I wish it didn't do and if if we remove to a to just like the the last headline and that's enough you're going away because it's discovery and every flexible 3 effects of and only and give some some some odd behavior this is our reference you I here that I made simplicity of use that only goal 3 workflows and found acting in viewing it was the here viewing of and this is the last this whole from from which you you you you the very very basic very plain but when you have something like that available on my mobile device that you aware that the and they can be quite powerful
but these are the so the inputs we take when we're doing tiling are for instance some you know full full reference system support of incoming and outgoing we we limit that is in the drop-down box there and you know we we limit that because of what we know 80 use what we we assume that users will be able to to what what what they will want you 3 5 7 typically in the 90 so the cases and then the the other if or for 3 to 6 the kind of you tacit support right now that becoming of the no actual rigorous support that will become a little bit later and again you can also how effective use 1 step the whole creating grating images you can do that and is tolerant to divide or you 1 have the flexibility having all those loose tile somewhere under system you can you can do that as well to 1 change from what you're making if you don't wanna make a huge PNG collection one's negative that collection of this oppression in the world that's a metadata at the bottom we once so then I have only actually
package inappropriate similar you can actually change the image of the you have as well when you're packaging you can take a PNG image because you want to say that transparency information and then converted to a defect is maybe you don't maybe spaces warm portal right you're targeting mobile device on the difference between your package that is there is PNG and J. Pei can be many hundreds of megabytes but the verifier verifiers job effects of and and a lot better I think of a you Y and then the Swing application you just saw unfortunately but but you re doing a an application we doing you for the applications we have only just solvers is is a large task the so a test for the following parts of the GeoPackage encoding center all the big ones that you have you know the 1 that we have in their speeches and features just because we haven't done if idea what looks like a Alaska asked to that you might you just saw was good you can you know all collapse that you can see the other of the divided in and what is past you can also see I apologize really blue
is the only is your that I had assessment of that it does show a warning on but I think this is a good illustration and that you have a warning you're not necessarily rated so for instance in this line here there's this evidence that you package shall contain but you 0 forces 5 year 3 1 you you can see that there wasn't a whole lot to layman right but on well the thing is if you have this was 1 byte header set in the binary file so it should be there but it's not you can still open it if not the so what did we learn
and buildings reference reference implementation and from what I've learned about about coding of learned about ourselves so overall for Java and I do I realize really early on that manages just very few choices for viewing but how this the mentioned in Europe that was really it and finding that was on some some arcane website that I found that had had a download link I wasn't sure of nowhere and that it is not for and so I wish I really wish we knew more about the job that that's actually found out about job fx from the last cluster gene instance inferences growing and also found out last at the meeting which which we from about that sooner what to use the number 2 and the part of mentation there but we have 4 tiling is still so far away the the fastest way to make tiles and we we we researched in Java new multithreading multiprocessing how how can we pack we speed this up the pipe the Python and implementation of using the Python multiprocessing library and it is so fast that you will kill your system you'll you'll use every single core and you will be usable but man who had ties really quickly we get we get we get developed our own right now and unfortunately he began as because the JVM of a woman limitations you have that and you'll find this a lot we knew this was last what you find this a lot when you're dealing with the database operations and optimizing the way you interact the database is to give you the biggest gains in performance and so reduce the massacre policymaking and in reducing the time to go into your of your Supply database so your database and in particular because that's gonna take the longest time right worrying about pieces of code and in certain areas where you using Lispers the hashmap like anarchy cup of improvement it person's going over a database the what elementary and compatibility for Android the for Desso out cells and the pressure like are you have job is for running and running Android devices will just work the the case of it's the 2nd 1 6 of officially 1 7 support is is there I believe you can make a 1 7 and Java project for bring it over it should work but we don't take that chance so the fact that we have devel 1 8 of and we had to go back for a a couple just a couple libraries to do that on the compatible if rendering that most so I so the desktop development for but for entries really ill-suited and the problem is when you're developing on a on a desktop for interacting the sequel like database those libraries knows that functionality that you have available to you is going to differ than it is on an Android device into a device having their own sequel light implementations debate and enjoyed whereas if you're on the desktop you have to use the bring that in that's not something that you just get the job and and then I will that 1 you know a buffered image in September the bitmap image on an Android Java implementation so you can have a have an adapter there to to help with that and and and necessarily very compatible and then with different testing framework that exist on on your J. universes in an Android 1 on Android and then there was a frustratingly no of globally available CMOS current into a jar available like me and so we do have a and after that source and references so regarding spec of you know I I thought there were grounds for 3 years there's there are still questions about spec still still different occasions on a single word and in this that's kind of frustrating at a given no they try to be as verbose and precise possible you still have different opinions and of and the ways that you can you can do your the you could Eutelsat so compromises key and and the dimension window can differ so in general think about them who you want to use your code and the and how and then constrain that lets you saw that the you I had it was actually gave you very very few options thank you posting FIL keep good thank you save and the
of any questions follow stated so 1 of the the 1st thing that you use the sure and so the good part sentences if you're if you're making a GeoPackage implementation of you will just acid GeoPackage what what extensions you at the end it has an extension that you don't have you still you still open because they get your your functionality and that you want you're not losing anything you just you get something with an extension that you don't support you can still look silicon that you still get all the things you would have the miller GeoPackage true if you if you have an extension of you that's going to be implementation-specific and that's that's kind of a shame but hopefully you register your extension with OGC URI registering with the body so that you can you can have people at the time that you know you get the use of possible and and now I'll help so there you have to it work the on the the of that you your I the use case was we we wanted fast base maps and fast data on a mobile device so but there there were you know you could add any tiles and it's tells straight and the has to work out however the more information we felt was better the more rigor you through around it so that's why we you yet you you have the the and the it's a that's a good question I think that GeoPackage there should exist outside of the mobile round and I think the mortgage is 1 in which that it works particularly well but there are plenty that don't work very well so for instance storing your data storing and how that data on the server in GeoPackage I would not recommend because you will have problems with sequel light at a certain point you eventually you get to a size in the size can be terrible of people right that's eventually get to a point where it serves no longer makes sense working with that you've economics and you might as well just having a big binary blog tiles or or file folder structure or something similar and and the is so the question well 1 minute if they have 1 question you thanks again for the presentation was very interesting and I want to see you mentioned that image compression I wanted to know about the other end of the the GeoPackage itself is there any associated over and and have you made any comparison to and B. times for example I there there's there's an but no difference as far as overhead from using Intel's GeoPackage that we've seen it's still if you if you want to be very fast just go directly to the task table then the contents and to start pulling tiles out and referencing enantiomers as you normally would there might be some some calculus again due to renumber entirely based on way the specs written however that's that's really so it's very fast image compression they're not really there to speed things up but rather than making smaller but that thank you again