GeoPackage and how open source is changing the way governments think about standards
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 183 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/32060 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Producer | ||
| Production Year | 2015 | |
| Production Place | Seoul, South Korea |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSS4G Seoul 201581 / 183
7
8
47
53
54
65
73
74
79
82
84
92
102
103
105
124
126
127
130
141
142
143
156
161
162
170
176
178
181
183
00:00
CuboidDependent and independent variablesMathematical analysisPlanningExtension (kinesiology)Greatest elementTesselationBitData storage devicePersonal digital assistantVector spaceMappingService (economics)DatabaseSequelSoftwareElectronic mailing listDifferential formSlide ruleEstimatorGoodness of fitCartesian coordinate systemGame theoryOpen sourceInformation securityTheoryCategory of beingMultiplicationMultiplication signReduction of orderLine (geometry)WordProduct (business)Projective planeNumbering schemeMereologyProcess (computing)Library (computing)Linear codeComputer fileGroup actionSingle-precision floating-point formatObservational studyoutputClient (computing)File formatDifferent (Kate Ryan album)Digital rights managementInstance (computer science)Integrated development environmentMobile WebStandard deviationInsertion lossCollaborationismAnalytic setSoftware developerWindowLink (knot theory)Android (robot)InformationMobile appScaling (geometry)Plug-in (computing)Computer animation
07:01
Level (video gaming)TesselationClosed setCartesian coordinate systemDecision theorySoftwareProduct (business)Goodness of fitDivisorMultiplication signCollaborationismOpen sourceTelecommunicationPresentation of a groupResidual (numerical analysis)Mobile appInformation securityConnectivity (graph theory)Electronic mailing listProcess (computing)Data conversionService (economics)RootLink (knot theory)Strategy gameQuicksortStandard deviationNumberOpen setContext awarenessDatabaseMedical imagingAndroid (robot)MultiplicationProjective planeSkalarproduktraumGraph coloringData storage devicePhysical systemLibrary (computing)Combinational logicDifferent (Kate Ryan album)CASE <Informatik>Single-precision floating-point formatVector potentialGroup actionNumbering schemeoutputJava appletCodeData exchangeWeb pageBitComputer animation
13:59
Open sourceDifferent (Kate Ryan album)Analytic setArithmetic meanSoftware developerRaster graphicsVector spaceSampling (statistics)Cartesian coordinate systemMobile appWeb pageElectric currentLibrary (computing)Right angleInformation securitySoftwareOnline helpStandard deviationMultiplication signData storage deviceDisk read-and-write headExtension (kinesiology)Process (computing)DatabaseInformationContext awarenessMathematical analysisKey (cryptography)Core dumpHand fanUniform resource locatorAreaBuildingOpen setArchaeological field surveyServer (computing)VelocityForestConservation lawView (database)Link (knot theory)Service (economics)Modulare ProgrammierungWordSequelMathematicsLatent heatMetropolitan area networkDivisorSelectivity (electronic)Forcing (mathematics)Figurate numberLevel (video gaming)
20:56
Computer animation
Transcript: English(auto-generated)
00:05
All right, well good morning, and thank you for coming. I'm Nathan Frantz, I am a project manager, and probably more importantly, a research geographer with the U.S. Army Corps of Engineers. I am going to be joined by Dr. Ben Tuttle from NGA, and I'll let him introduce himself a little bit more when he gets there, gets up here.
00:23
But for the most part, I want to kind of tell a story about how government agencies are starting to adopt this idea of open source software. And obviously, it should be known that it's slowly, and it's for security purposes, but what we're finding is obviously the gains that we have by actually doing that.
00:43
So it's not only what we're getting out of it, but of course, what we can put back into it. Anything that the government funds, of course, is at least at heart already kind of owned or property of U.S. citizens. So it's not that big of a leap to say, well, if it's the U.S. citizens, then why not give it back to the community?
01:00
And so that's what we're trying to do. And we play this game of where that line is. So obviously, we have security issues. I'm not gonna talk about those today. And then we have the ability to actually contribute back to community, get all this information and all this data and all this software back that we can then leverage, and it really is driving innovation. And that innovation that we've seen
01:22
has been in the mobile handheld community mostly, which is why I want to talk about Geopackage. So before we do that, let me give you a quick background. So Geospatial Research Lab, it's one of seven labs within the ERDCs, or excuse me, within the Army Corps of Engineers and Engineer Research and Development Center, which of course is the R&D side of the Army Corps of Engineers.
01:41
Being the Army Corps of Engineers, we do a lot of humanitarian assistance and disaster response work. And I just wanted to kind of put that plug in because that's a big reason why we use a lot of mobile handheld technology now. So whenever you hear me kind of give examples, think of that humanitarian response or disaster response. And that's where we use a lot of our technologies for mobile handheld.
02:01
If you want to find out more about us, of course the link is at the bottom. So let me jump in, and you know, I titled this slide Why Geopackage, but really it's this kind of wide data standard. So being a government agency, being the US government, you know, we're big, we're big. There's a lot of people, a lot of players, and a lot of customers. And so we have tremendous issues
02:21
of trying to provide technologies and even data and software to this large community. And so, you know, especially within the US military, so we have this large scale collaboration issue. And having so many different government agencies with so many different customers, it's very difficult for us to actually communicate
02:43
and put all of that technology together. And so what we've also found is that actually using the open source community, we actually communicate a little bit better in some instances, because we actually have a single place to do it. And so that's one of the things we've seen. Again, this is kind of based around this idea of disaster management.
03:01
And the reason I keep bringing that up is because we actually have to operate a little bit differently than a lot of the commercial, the commercial side does. And that's because typically what we deal with are disconnected environments. And so if you think of mobile handheld, they are connected devices. They run on services. That's what they do. That's how they were designed.
03:20
That's how Android was designed. That's how Apple builds its software. They run on services. And when those go away, the device kind of becomes a brick. Or at least they did in the early days. It's getting a lot better. So what we wanted to do with Geopackage, and what we want to do with data standards, was to create a way that we could actually kind of have data at rest to start with,
03:41
but then actually have kind of more of a services at rest as well. And so what we want to do is not just store data so we can visualize data on the device. We actually want to run analytics on the device, which is where a lot of this got started. Also, of course, as we just heard, obviously the explosion of the mobile handheld and the application development process.
04:01
So in the early days, back in 2009, 2010, when Ben and I first met, the application, you know, this whole idea of application development for ThinCliner for mobile handheld devices was really taking off. Everybody that had ties to the US government was throwing apps at us left and right. The Army in 2010, I do believe,
04:21
had an apps for the Army competition even, available and open to all government agencies. And so we actually competed in that. And so what happened was kind of the Good Idea Fairy came in and sprinkled all these different data sources and software applications on us, and none of them communicated. None of them used the same data. And that was a big problem.
04:40
That was the first thing we noticed. And that's kind of where Geopackage got started, was this idea of to reduce data duplication and increase data interoperability. Really that simple. So Geopackage then was designed for the mobile handheld device, so Thin Client. Excuse me, it's SQLite-based,
05:01
and it was really designed initially for tiles. We were really interested in tiles, and of course we also wanted to bring in some vector data. But for the most part, Geopackage is a single file, SQLite-based, that stores raster, tiled maps, vector data, a manifest, a self-describing manifest, so the software knows what's in it,
05:21
and then also has the ability to have extensions, which again is key for what we're trying to do here, because we didn't just want to put the data on the device, this idea of data at rest, we want to have our services at rest as well. We want to be able to do analysis on the device. So those extensions come into play, and we'll talk about some of that later. So a brief history, I talked,
05:40
so Ben and I started working together about five years ago, and in the early days, we were big on MBTiles, for a lot of the raster-tiled products, we were big on SQLite, Spatialite. Spatialite, of course, gave us the ability to actually do some analysis, some SQL calls on the database themselves for vector data, and we have some initial tools
06:02
that would allow us to do some analysis with the device. And that was all well and good, but again, it goes back to our problem, within the U.S. Army at least, of this idea of data duplication. It wasn't just storing the data, especially tiles, on the device multiple times, it was this idea of also duplicating
06:21
the production of that data. I cannot tell you how many times we've tiled the same products in different formats or in different tiling schemes, and that was a tremendous problem, or even in a different projection. And I don't want to get into projections too much, but you would be amazed what just changing a projection does to the U.S. military. It slows us down.
06:42
So after that, what we did is we actually said, all right, we need to figure this out. So Ben and I kind of kicked this off in the early days. We tossed it over to my sister agency, the Army Geospatial Center, and said, you guys kind of need to take this and run with it. They got together a group, a working group, and they created some Google groups as well. So that was kind of our first look
07:02
at actually putting this out in the open community, let people come in and weigh on what a potential data, data at rest would look like for mobile handheld. And from there, we quickly realized that there needed to be some sort of official standard to do this, because everybody, again, still had, you know, this is my idea, this is what works, MB Tiles, what have you,
07:22
and it's certainly nothing wrong with MB Tiles. It's a fantastic product. The problem is it didn't necessarily work for us at the time. And so we put it together as a standard working group. We tossed it over the fence to the Open Geospatial Consortium. They actually made this an official spec in 2014, so it took a couple years, a little slow.
07:40
But now that it is, we can actually go back and fall on this and say this is a standard that's blessed by the open community, and it's something that we as the U.S. Army can actually pick up now and use. So in the past, it was, well, MB Tiles is really good, it's kind of a de facto standard, and you can say the same thing about Shapefire or KML. So that's what we're gonna do, that's what we're gonna use, and that's what's gonna happen.
08:00
The problem is is not everybody's gonna make that same decision, not every agency's gonna make that same decision. And again, we go back to this idea of having nothing that communicates, and our applications are actually talking to each other or looking at the same data. So we're storing the data twice, three times on the same device.
08:23
So really what it led to in those early days was this idea of collaboration. And it's not something that we're always the best at. And again, a lot of it has to do with security issues. But I just wanted to throw this up, and this is just a short list. There's many more industry and government agencies that are working on Geopackage. But a lot of this work was done
08:40
to the advantage of each individual company or each individual agency. There wasn't a whole lot of money passed around to get Geopackage developed or pushed through AGC. A lot of it was done on the individual dime of either the agency or the government itself. And to me, that was unprecedented within how we typically had done things.
09:03
Typically as a government, we pay a commercial vendor to bill us a product, and in that product comes a data standard that they developed with it. And so this was very different. And we're trying to replicate that as we move forward because of course Geopackage is just for mobile handheld, we want to expand on that services
09:20
all the way up to the national level of actually creating data and storing data, geospatial data of course. So where we were. So let's go back to 2010 when we first started. A lot of different applications, a lot of those applications did the same exact thing. The good news is most of those applications were actually developed on free and open source software.
09:41
So that was good. So we could at least modify them and move the code around. But the problem was they all relied on their different databases. And even if they used the same databases, let's say two applications use the same MBTiles database, we could still have projection issues or tile scheme issues and everything fell apart. So again, we were storing the data multiple times on the device. And again, we were also creating the data multiple times.
10:04
And that's a big no-no for us just because of the size of the data that we have to be able to manage. So what we ended up doing in the early days was have glue code. We'd have glue code that would either do data exchange between the applications or it would actually
10:21
help the applications communicate with each other or push the data around. So this glue code was the big problem in the early days. So that's when we kind of went through this whole idea of Geopackage, starting pushing the idea. On the backside, because it does take a while to get a standard through OTC, we also try to reduce even just the number
10:42
of data standards that we're using outside of Geopackage as it was moving along. And so where do we want to be? Where do we want to take this? What we want to do is ideally is we'd have a combination of an open source and proprietary applications. Obviously the applications aren't going to go away. They use a single database. That would be an ideal situation for,
11:03
in my case, the US Army, right? And so it's a difficult process, it's a long road, but again, what we found is if we can leverage the open source community, we can actually get ground support and push this as opposed to the Army saying, thou shalt not use MB Tiles, thou shalt use Geopackage. The problem is whenever MB Tiles makes a new product,
11:23
it's innovative, people are going to use it, right? It's going to happen. And so what we're trying to do is we're trying to be innovative at the data level with the open source community, and we're trying to be innovative at the application level with either closed software, proprietary software, as well as open source software. I think that that's something that we can get to
11:41
with this model versus just going out and paying vendors to build these applications for us and then bringing their own data solutions. Okay, so with that, I'm going to actually turn it over to Ben, and he's going to talk about some of the ongoing R&D with Geopackage and some of the open source software that they've been developing.
12:02
Thanks, Nathan. So special thanks to Nathan for letting me jump in on his presentation here. So to kind of tie this to where Nathan's been going with this whole story, so at MGA in the last few years we've started to convince people, right, that just using open source is not really supporting open source. So we've sort of shifted the conversation now
12:22
to doing things where we're actually funding some of the companies that are working on the open source packages, so contributing back that way. And then more recently realizing that a lot of the stuff that we're developing in-house, oftentimes we can modularize out components of that that are reusable, and we've gotten through the whole process now where we've got our own GitHub page
12:40
and we can throw stuff up on there. We have to go through all sorts of crazy legal review with our lawyers and things to do it, but it's all doable. And so we've got a number of things out there. You can see the root of our account on GitHub up there in the links if you want to go check it out. There's a whole bunch of things besides just the mobile stuff. But so one of the things we've done recently
13:00
is we were looking at how to get Geopackage now that it's out there, and it's a standard to start building it into some of the mobile apps that we have, and how to get other people to make it easier for them to build it into their mobile apps. So we had a guy from Bit Systems that we were working with who started going through this and wrote this set of libraries that we put out earlier this year to make it so you can do
13:22
a read-write of a Geopackage on Android. It's all just Java, so there's also some components that are not really Android-specific that you can just pull out and run in a Java app. And then he's wrapping up iOS as well right now, so that'll come out on GitHub probably a few weeks, maybe six weeks, depends on, again,
13:40
that lawyer discussion I brought up. But that'll be out there. It's all MIT license, so anybody who wants to grab that and reuse it, that's kind of what we wanted to get at, is putting that out and really write that two-fold thing of we're not just using it anymore, but we're giving back through trying to fund things and then also trying to use our developers
14:01
to contribute back to the community as well. So if anybody does want to check that out, there is a sample application on our GitHub page that puts all those libraries together so you can see how they work. And so those are just some screenshots of that sample app where you can browse through the Geopackages that are on your device and then render those on a map. These are some of the Geopackages that are available
14:21
on Geopackage.org, so we've been using those for testing. So that's gonna be a big thing, right, that I know Nathan and I have been looking at is these start getting created by many different software packages, making sure that that's all supportable. But yeah, so please go check out our account if anybody's interested, wants to fork it,
14:41
wants to do a pull request. Love to see that happen. With that, I'll let Nathan keep going. Thanks, Ben. So to go back to the Geospatial Research Lab and some of the things we're doing, the biggest thing for us is it's gonna be very difficult for us to be able to push a lot of our application development out to the open source community.
15:01
Again, a lot of it has to do with security issues. And so we know off the bat that's gonna be very difficult. It doesn't mean we're not trying, and it doesn't mean that we're not leveraging current open source technologies, because we are. But what we see is quick wins with actually getting a lot of our technologies to actually, on the data side, to be open sourced.
15:21
So a lot of our data is proprietary to us, or for security reasons, not releasable. However, the way we store that data, the way we access data, the way we manage that data, all of those process can pretty easily be actually pushed out to the open source community, especially now that Geopackaged is an OGC spec. And so what we're doing is we're actually developing
15:41
some tools for creation and dissemination, or really just management, Geopackaged management, because that's a tremendous issue for us right now. Obviously, all these Geopackages start floating around, all these SQL-like databases start floating around. What's in them, how do you manage them, and where does that data go, where does that data die, where does that data live?
16:01
And so what do we do with it? If you want to find out more about what we're doing with that, and that information is also out on GitHub right now, you can go to our GitHub links, and you can also actually head to a Geopackaged discussion tomorrow by Mr. Steve Lander, who's gonna talk about some of those capabilities, because he's the one that's actually developing them.
16:21
So, I mentioned extensions early on. This is big to us, this is really key. Again, getting data out to the end user is critical. So that's kind of the situational awareness piece. However, again, we still don't have access, or we can't leverage services, so how do we do things like analysis on the device? We're working on extensions. So just like spatial light,
16:40
I'm a big fan of spatial light. We still actually use it in a lot of our capabilities. What we want to do is we want to take some of those same key or core capabilities within spatial light and move them over to Geopackage. And so the idea isn't necessarily to duplicate spatial light, but the idea is to actually, again, just be able to do some of these analysis capabilities and build from there. And hopefully we can kind of work back and forth
17:01
with the spatial light community as we do that, because I certainly don't want to shortchange anything they're doing. The big thing for us is routing. So how do you do routing? How do I get from a known location to a potential area where I need to go survey with my GeoPaparazzi app? So how do I navigate to that location? Again, no services, right?
17:21
So how do I do it? I've got to have all the data on the device. I've got to have all those analytics on the device, and I have to have a means to do that. So we're actually working on that. Right now, Geopackage is two-dimensional, so we want to bring in some elevation data. We want to get this thing to where we can potentially have 3D data for vector and raster data. There's also,
17:40
and I'll just kind of skip through a couple of these, there's also no symbology or styling, which I know is a big problem, so we're going to work on that as well. And we're trying to figure out how to do that again. My hope is this community can help us with that, because it's not something that we're particularly good at or have an understanding of. So we are working on that.
18:02
So how do we move forward? And I'm pretty much at time, so I'm just going to skip right to the end. The thing with Geopackage, and there's a thing with the open source community, with Geopackage, for Geopackage, excuse me, we were very reactive to emerging technologies.
18:21
So any innovation that occurred within the army, within the military, occurred internally and it stayed internally. So a lot of times we were reactive to emerging technologies from the commercial world. Of course, that is where a lot of technology comes from, right? So we were reactive to whatever they were doing. They were doing it, we got to figure out a way to incorporate it, because our troops want it,
18:41
our soldiers want it, what do we do? And we have to fix it. What we're trying to do is be more proactive through a lot of this open source community, but also to help try to drive standards forward. Not just Geopackage, but any standard, help drive it forward, help say our piece, and that's what I'm going to ask of you guys. Geopackage is an open standard right through OGC,
19:00
so if there's something you don't like in Geopackage, ask to have it changed. Tell us. If we don't know, we're not going to change it. And I can tell you there were a couple big players as Geopackage worked its way through OGC that were very loud voices, and of course the squeaky wheel gets the grease, right? So if you're not saying something, if you're just saying, ah, that doesn't really work for me,
19:21
let us know. Tell OGC, tell us, put it on a board, and hopefully we can kind of move forward with that. So it's definitely a community-driven effort. And with that, I'm done. Very little over time, so I'll take questions quickly, and then turn it over. In the back, yes sir.
19:49
Yeah, so I'm not the best answer to that question, I'll be completely honest with you. I kind of live in my world. And you know, it's the truth, I live in my world. If you go to Geopackage.org, you can look at all the companies
20:00
that are developing with Geopackage, and there's a lot of sample applications that are out there. LUCIAD has been a big player. Skyline is developing stuff. Reinventing Geospatial, the guys that work with us have developed a lot of stuff and put it up on GitHub, and on the Geopackage website, NGA. It's been slow moving, to be completely honest with you. I'm not particularly happy with where it is.
20:22
I was hoping we'd be further along as far as community involvement and adoption of Geopackage, but we're gonna keep pushing it. I mean, right now, because it's an official OGC spec, and it's something the Army can hang their hat on because of that specification, we're gonna keep pushing it, and we're gonna keep using it for a while. So it's not gonna go away in our world,
20:41
and we're hoping that we can, you know, take and leverage anything we do and push it out to the community so everybody else can. Any other questions? Okay, I appreciate it. Thank you.