GDAL Tips and Tricks
Formal Metadata
Title |
GDAL Tips and Tricks
|
Subtitle |
GDAL installation, Python usage, and Cloud GeoTIFFs
|
Title of Series | |
Author |
|
License |
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. |
Identifiers |
|
Publisher |
|
Release Date |
2018
|
Language |
English
|
Content Metadata
Subject Area |

00:00
Goodness of fit
00:38
Moving average
Point cloud
Bit
Online help
Geometry
01:01
Image warping
Statistics
Multiplication sign
Computer-generated imagery
File format
Grass (card game)
Shape (magazine)
Raster graphics
Medical imaging
Different (Kate Ryan album)
Utility software
Library (computing)
Information
File format
Projective plane
Image warping
Bit
Shape (magazine)
Open set
Equivalence relation
Mathematics
Vector space
Raster graphics
Abstraction
Library (computing)
02:35
Installation art
Keyboard shortcut
Repository (publishing)
Maxima and minima
Set (mathematics)
Online help
03:27
Revision control
Multiplication
Computer file
Different (Kate Ryan album)
Configuration space
Virtualization
Gamma function
Virtual reality
Optical disc drive
Physical system
Physical system
04:15
Area
Keyboard shortcut
Validity (statistics)
Transformation (genetics)
INTEGRAL
Structural load
Disintegration
Keyboard shortcut
Image warping
Bit
Instance (computer science)
Raster graphics
System call
10 (number)
Area
Power (physics)
Process (computing)
Vector space
Raster graphics
Calculation
Library (computing)
Geometry
06:30
Scripting language
Torus
Image warping
Slide rule
Scripting language
Electric generator
Computer file
Validity (statistics)
Tesselation
Source code
Set (mathematics)
Virtualization
Instance (computer science)
Calculus
Raster graphics
Medical imaging
Virtual reality
Different (Kate Ryan album)
Buffer solution
Utility software
Optical disc drive
Geometry
08:48
Reading (process)
Operator (mathematics)
Projective plane
Maxima and minima
Optical disc drive
09:10
Laptop
Web page
Reading (process)
Presentation of a group
Server (computing)
Link (knot theory)
Multiplication sign
Range (statistics)
Source code
Execution unit
Device driver
Raster graphics
Open set
Average
Wiki
Medical imaging
Software testing
Data compression
Point cloud
Area
Email
Information
Mapping
File format
Tesselation
Digitizing
Range (statistics)
Device driver
Uniform resource locator
Process (computing)
Internet service provider
Point cloud
Website
Probability density function
13:14
Email
Link (knot theory)
Code
Projective plane
Electronic mailing list
Online help
Bit
Mereology
Wiki
Website
Utility software
Series (mathematics)
Reading (process)
14:51
Rounding
Slide rule
Presentation of a group
Computer file
View (database)
Zoom lens
Collaborationism
Set (mathematics)
Open set
Twitter
Atomic number
Different (Kate Ryan album)
Energy level
Scripting language
Computer icon
Email
Software developer
Projective plane
Planning
Bit
Instance (computer science)
Website
Point cloud
Right angle
Freeware
Geometry
Library (computing)
00:05
adios good it's on
00:25
you you
00:35
I'm gonna talk about I'll first give a
00:38
bit of an introduction talk about some tips when installing jido how to use it with Python some random tips and then a bit about how to use these cloud optimized and how to make cloud optimized geo tips and also where to go for some help so there's a quick
00:59
introduction about myself I'm the team
01:01
lead for today pipeline team in Berlin for playing labs Germany I've been there
01:05
since 2014 working for blackbirds and we were acquired by Planet I am Belton but I was raised in Florida so that explains the accent of it this is my fourth year attending Fossum and the first time speaking so a bit about GTO who here has used G tall or is using G go already alright good so I'll just quickly go over this so G dollars the geospatial data abstraction library I was initially created by Frank warmer damn he's actually my colleague back in San Francisco the project is damaged by osg oh it's a c e c++ library can read a whole bunch of vector so shapes and raster or image formats there's a whole bunch over 150 raster's and 90 vector formats it's as you heard it's kind of like the base of everything that like they say every geospatial project uses this internally so something like most common GTL utilities are like Chitale warp 4 if you're like jim was aching reprojection and warping g'd all translate to change from one raster format to another and then the vector equivalent of ogr to OG r and then GL info where you know you can not just get the information but you can also generate a whole bunch of statistics out of that and get it in a whole bunch of different formats and so in on the insulation side on mac is pretty easy
02:37
there's a if you use homebrew which you
02:41
should there's the OS do for mac tap that you can use it's also managed by OS geo and they're always looking for some help there and then you just install the G del 2 and G talk to python bindings on a bun to there's a about to das PPA that has G da there but there's also a cross-platform way of doing this so if you're already using anaconda you basically just kinda install with jido and that kind of like sets up a lot of the dependencies that you need there but I don't really recommend doing either of those because you get a lot of dependency problems so
03:32
planet we use a lot of VMs and
03:34
containers to isolate all the system dependencies to basically make it so you can just rebuild the whole system and get back to a working configuration you don't have to worry about like upgrading something for some other package that you might be using and breaking your whole jido insulation and it also kind of lets you avoid having to use the Python virtual lens where you also get some problems if you're trying to do like multiple versions of GL and different virtual environments that just gets very messy and basically you can just put the setup instructions in docker file wagon file or use an existing set up to kind of get started and then just build up there with whatever dependencies you might need so
04:17
working in Python G doll two gave us a
04:22
lot of powerful Python bindings so you don't need to use the command line call anymore you can just basically put the command line arguments in the Python called Vegeta wart for instance and there's also I was reading a bit about this at the OG o - o gr for instance it's not called øg øg are in the Python bindings but is jino vector translate there's a whole RFC about how this was set up but yeah it makes it a lot easier to have that and you don't have to worry about the full sub process call also there's a very good numpy integration so if you are doing things like windowing you can actually just read the array and specify the offset that you want and then you get full numpy array you can do all the vector and sorry all the raster computations that way as well also if you are just kind of trying to do some small things you don't actually have to use the detail Python bindings directly there's a few good libraries that we use as well rest here you or Rosario from Matt Fox it's basically a pythonic version of G Dahl so it's all jido under the hood but it's basically just a Python API that's pythonic and it's a lot easier to use when you're wanting to open you can do a lot with that for vectors and shapes shapely and Fiona and the same person it's also pythonic shapely only has geo spend see if I remember correctly and then Fiona has full geology our dependency and of course they're also a lot easier to use you can load GJ songs and wkts and all this stuff and do your validation and intersection area calculation and all these other transformations so some other tips what
06:36
I really recommend is that if you are wanting to learn a lot more how to use jeetil especially like the C++ API there's a lot of the utilities that are some of them that I mentioned in previous slides that you can just read their source code try to understand how they work and then that gives us basically a bit better than the documentation on how to use things especially if you're looking at GL warp that shows you how to actually do all the things that it does under the hood as well as there some of them that are Python scripts so the g-dog calc for instance general calc I that does all the loading into the numpy array and does the for instance if you want to do like a difference then you can just look at how it extracts the operations and execute them so that's basically my first tip on that also if you're working with geometries that you get from random places or that you're generating sometimes especially around Dateline you get some weird issues where the the geometry doesn't validate anymore and basically you can just apply a zero buffer on it and then that ogr kind of magically makes it a valid geometry so really helpful whenever you're having weird problems with geometries just apply a buffer to it with actually no buffer but that makes it work there's also VR T's so VR T's are virtual data sets and you can basically treat a whole bunch of imagery tiles as like one big image you can also basically take a big image and chunk it up into smaller files and then join it all back together in this VR T which just references all the images and where they should be in if they were one raster oh yeah so VR T's are just
08:56
basically XML files that describe all the operations and they do need everything to be in one common projection so just keep that in mind if you are trying to do that the other
09:13
really cool thing is that a lot of imagery providers are now making how to optimize do tests so what's the cloud
09:22
optimized use it so these are geo tips that work a lot better for cloud processing so essentially what they do is they have like the header information in the very beginning and it's a tiled image so usually 256 by 256 or 512 by 512 tiles and they also have all the overviews built in sorry by doing this we you can essentially just do a range request so you can get the header or jido looking basically get the header and then figure out what bike range the imagery that you need if you're doing a window or where the overviews are if you're just getting some if you just need any overview of the image and then it downloads only the data that it actually needs so if you're using the vsi curl driver in Gino you can basically just put the URL in there and of the image if you're getting it from open area map digital globe planet the Landsat 8 bucket on AWS is also in this format and there's a whole bunch more Co giotto org has like a lot more information about this then you can really save on the download times and not having to download the whole raster just to get the small piece that you need if you are generating image and you want this to be in the cloud optimized USA format there's basically two things that you need to do if you don't already have overviews builts then you could just use gino add oh and overviews to build the overviews so basically just put your image where it says in TIFF and run this should be pretty quick and that actually modifies the image and puts it indirectly if you are doing an external overview it's not going to work the second thing is to then do GDL translate where you basically tell that you want it to be tiled and you want to copy the overviews that were in there and then this recommends to use lzw compression you can also use deflate but there's some issues some some things aren't fully compatible with deflate and and the other important thing is that if you're when you're hosting it the server needs to support the range request so HTTP 1.1 if you're doing if you're just putting it on s3 or Google Cloud or Azure they all support this byte serving as it's also called to use the cloud optimized unit if basically if you're using the VA site kernel driver that's all under the hood handle magically some examples are on the G dal wiki there's also a good basically Jupiter notebook that describes that and slinked here I also have the presentation uploaded on the Boston website so if you look up the page for this presentation you'll see the presentation link there's a PDF so here's an example of using the via site curl driver as you can see it's basically just doing the slash a sidecar slash the URL so this one is Landsat 8 image on the glanced at it alas pocket which is a great open data source if you want to start playing around with raster and need some imagery so where to get
13:17
help this is also a very important topic
13:20
but the GI Stack Exchange which I think I saw a shirt there that's a really good place you'll find a lot of people from the developing community Frank and I both answer questions on there there's a jido IRC channel as well and the Udall dev mailing list is a great way to get answers you know or the main website has you know the wiki and everything else so you can really get like all the documentation on there but another great resource is there's a series of medium posts by a colleague Rob Simon where he goes over really introduction to G dal you know a bit about the history of projections and whatnot so it's a really good read like three part series there's also the Python geology our cookbook I've used it quite a lot when starting out would you dull to figure out how to do things in Python doing that and the reading the general utility code as well those two things were important and I also made one of these awesome geospatial lists on github so a bit of shameless self-promotion of that there and yeah so basically these people
14:54
helped out with the presentation mostly some my colleagues and Evan who's basically one of the main developers of G Dahl now and all the imagery here is from my company so all the beautiful pictures you saw on the slides the we also have an open data set of a California so it's Creative Commons Licence imagery from like two weeks ago or something like that or you can basically just go on plan website and make an account and get some data if you want to have some more free data sets that you want to use for your stuff and that's it just picture my cat and thank you and you can reach me on Twitter github you can send me an email and that's it thank you any questions yes no this has been supported indeed all for a while actually so the cloud optimized geo tips are nothing really new the it's more like leveraging existing features of G doll I'm kind of like a bit more standardized way so so the question is when you have all these different libraries and you they all recommend to use docker but they have their own docker containers and you want to basically put them together we kind of have had the similar problem of doing that we rely heavily on GTL but of course there's other libraries that you need to use and basically one of the things you can do is to kind of try to glue these different doctor files together so to like look into it and build it that way we also rely heavily on the atoms that planet and so basically we have like a big script that you know as you start trying to integrate these different projects you kind of build up a VM and if you start going down the really weird path then you can kind of still recreate the whole VM back out so it is a lot of work unfortunately but yeah the the VM makes it really like sure that like you can always back out and kind of start over and that's kind of like your source of true if it works on the fresh PM then you just use that but if you're like doing something weird then you can yeah so I don't think Yugi is fully supports it yet I think in qgis 3 which should be released sometime soon they will have some of that support but yeah it's not fully supporting TGS yet so the question was our Kyle optimized gos if supported QGIS correct yeah you could once it's supported then basically it's streaming the data right and especially if you're viewing it at a higher zoom level for instance it'll just use the over views instead of downloading the full roster any other questions all right [Applause]
