adios good it's on

you you

I'm gonna talk about I'll first give a

bit of an introduction talk about some tips when installing jido how to use it with Python some random tips and then a bit about how to use these cloud optimized and how to make cloud optimized geo tips and also where to go for some help so there's a quick

introduction about myself I'm the team

lead for today pipeline team in Berlin for playing labs Germany I've been there

since 2014 working for blackbirds and we were acquired by Planet I am Belton but I was raised in Florida so that explains the accent of it this is my fourth year attending Fossum and the first time speaking so a bit about GTO who here has used G tall or is using G go already alright good so I'll just quickly go over this so G dollars the geospatial data abstraction library I was initially created by Frank warmer damn he's actually my colleague back in San Francisco the project is damaged by osg oh it's a c e c++ library can read a whole bunch of vector so shapes and raster or image formats there's a whole bunch over 150 raster's and 90 vector formats it's as you heard it's kind of like the base of everything that like they say every geospatial project uses this internally so something like most common GTL utilities are like Chitale warp 4 if you're like jim was aching reprojection and warping g'd all translate to change from one raster format to another and then the vector equivalent of ogr to OG r and then GL info where you know you can not just get the information but you can also generate a whole bunch of statistics out of that and get it in a whole bunch of different formats and so in on the insulation side on mac is pretty easy

there's a if you use homebrew which you

should there's the OS do for mac tap that you can use it's also managed by OS geo and they're always looking for some help there and then you just install the G del 2 and G talk to python bindings on a bun to there's a about to das PPA that has G da there but there's also a cross-platform way of doing this so if you're already using anaconda you basically just kinda install with jido and that kind of like sets up a lot of the dependencies that you need there but I don't really recommend doing either of those because you get a lot of dependency problems so

planet we use a lot of VMs and

containers to isolate all the system dependencies to basically make it so you can just rebuild the whole system and get back to a working configuration you don't have to worry about like upgrading something for some other package that you might be using and breaking your whole jido insulation and it also kind of lets you avoid having to use the Python virtual lens where you also get some problems if you're trying to do like multiple versions of GL and different virtual environments that just gets very messy and basically you can just put the setup instructions in docker file wagon file or use an existing set up to kind of get started and then just build up there with whatever dependencies you might need so

working in Python G doll two gave us a

lot of powerful Python bindings so you don't need to use the command line call anymore you can just basically put the command line arguments in the Python called Vegeta wart for instance and there's also I was reading a bit about this at the OG o - o gr for instance it's not called ?g ?g are in the Python bindings but is jino vector translate there's a whole RFC about how this was set up but yeah it makes it a lot easier to have that and you don't have to worry about the full sub process call also there's a very good numpy integration so if you are doing things like windowing you can actually just read the array and specify the offset that you want and then you get full numpy array you can do all the vector and sorry all the raster computations that way as well also if you are just kind of trying to do some small things you don't actually have to use the detail Python bindings directly there's a few good libraries that we use as well rest here you or Rosario from Matt Fox it's basically a pythonic version of G Dahl so it's all jido under the hood but it's basically just a Python API that's pythonic and it's a lot easier to use when you're wanting to open you can do a lot with that for vectors and shapes shapely and Fiona and the same person it's also pythonic shapely only has geo spend see if I remember correctly and then Fiona has full geology our dependency and of course they're also a lot easier to use you can load GJ songs and wkts and all this stuff and do your validation and intersection area calculation and all these other transformations so some other tips what

I really recommend is that if you are wanting to learn a lot more how to use jeetil especially like the C++ API there's a lot of the utilities that are some of them that I mentioned in previous slides that you can just read their source code try to understand how they work and then that gives us basically a bit better than the documentation on how to use things especially if you're looking at GL warp that shows you how to actually do all the things that it does under the hood as well as there some of them that are Python scripts so the g-dog calc for instance general calc I that does all the loading into the numpy array and does the for instance if you want to do like a difference then you can just look at how it extracts the operations and execute them so that's basically my first tip on that also if you're working with geometries that you get from random places or that you're generating sometimes especially around Dateline you get some weird issues where the the geometry doesn't validate anymore and basically you can just apply a zero buffer on it and then that ogr kind of magically makes it a valid geometry so really helpful whenever you're having weird problems with geometries just apply a buffer to it with actually no buffer but that makes it work there's also VR T's so VR T's are virtual data sets and you can basically treat a whole bunch of imagery tiles as like one big image you can also basically take a big image and chunk it up into smaller files and then join it all back together in this VR T which just references all the images and where they should be in if they were one raster oh yeah so VR T's are just

basically XML files that describe all the operations and they do need everything to be in one common projection so just keep that in mind if you are trying to do that the other

really cool thing is that a lot of imagery providers are now making how to optimize do tests so what's the cloud

optimized use it so these are geo tips that work a lot better for cloud processing so essentially what they do is they have like the header information in the very beginning and it's a tiled image so usually 256 by 256 or 512 by 512 tiles and they also have all the overviews built in sorry by doing this we you can essentially just do a range request so you can get the header or jido looking basically get the header and then figure out what bike range the imagery that you need if you're doing a window or where the overviews are if you're just getting some if you just need any overview of the image and then it downloads only the data that it actually needs so if you're using the vsi curl driver in Gino you can basically just put the URL in there and of the image if you're getting it from open area map digital globe planet the Landsat 8 bucket on AWS is also in this format and there's a whole bunch more Co giotto org has like a lot more information about this then you can really save on the download times and not having to download the whole raster just to get the small piece that you need if you are generating image and you want this to be in the cloud optimized USA format there's basically two things that you need to do if you don't already have overviews builts then you could just use gino add oh and overviews to build the overviews so basically just put your image where it says in TIFF and run this should be pretty quick and that actually modifies the image and puts it indirectly if you are doing an external overview it's not going to work the second thing is to then do GDL translate where you basically tell that you want it to be tiled and you want to copy the overviews that were in there and then this recommends to use lzw compression you can also use deflate but there's some issues some some things aren't fully compatible with deflate and and the other important thing is that if you're when you're hosting it the server needs to support the range request so HTTP 1.1 if you're doing if you're just putting it on s3 or Google Cloud or Azure they all support this byte serving as it's also called to use the cloud optimized unit if basically if you're using the VA site kernel driver that's all under the hood handle magically some examples are on the G dal wiki there's also a good basically Jupiter notebook that describes that and slinked here I also have the presentation uploaded on the Boston website so if you look up the page for this presentation you'll see the presentation link there's a PDF so here's an example of using the via site curl driver as you can see it's basically just doing the slash a sidecar slash the URL so this one is Landsat 8 image on the glanced at it alas pocket which is a great open data source if you want to start playing around with raster and need some imagery so where to get

help this is also a very important topic

but the GI Stack Exchange which I think I saw a shirt there that's a really good place you'll find a lot of people from the developing community Frank and I both answer questions on there there's a jido IRC channel as well and the Udall dev mailing list is a great way to get answers you know or the main website has you know the wiki and everything else so you can really get like all the documentation on there but another great resource is there's a series of medium posts by a colleague Rob Simon where he goes over really introduction to G dal you know a bit about the history of projections and whatnot so it's a really good read like three part series there's also the Python geology our cookbook I've used it quite a lot when starting out would you dull to figure out how to do things in Python doing that and the reading the general utility code as well those two things were important and I also made one of these awesome geospatial lists on github so a bit of shameless self-promotion of that there and yeah so basically these people

helped out with the presentation mostly some my colleagues and Evan who's basically one of the main developers of G Dahl now and all the imagery here is from my company so all the beautiful pictures you saw on the slides the we also have an open data set of a California so it's Creative Commons Licence imagery from like two weeks ago or something like that or you can basically just go on plan website and make an account and get some data if you want to have some more free data sets that you want to use for your stuff and that's it just picture my cat and thank you and you can reach me on Twitter github you can send me an email and that's it thank you any questions yes no this has been supported indeed all for a while actually so the cloud optimized geo tips are nothing really new the it's more like leveraging existing features of G doll I'm kind of like a bit more standardized way so so the question is when you have all these different libraries and you they all recommend to use docker but they have their own docker containers and you want to basically put them together we kind of have had the similar problem of doing that we rely heavily on GTL but of course there's other libraries that you need to use and basically one of the things you can do is to kind of try to glue these different doctor files together so to like look into it and build it that way we also rely heavily on the atoms that planet and so basically we have like a big script that you know as you start trying to integrate these different projects you kind of build up a VM and if you start going down the really weird path then you can kind of still recreate the whole VM back out so it is a lot of work unfortunately but yeah the the VM makes it really like sure that like you can always back out and kind of start over and that's kind of like your source of true if it works on the fresh PM then you just use that but if you're like doing something weird then you can yeah so I don't think Yugi is fully supports it yet I think in qgis 3 which should be released sometime soon they will have some of that support but yeah it's not fully supporting TGS yet so the question was our Kyle optimized gos if supported QGIS correct yeah you could once it's supported then basically it's streaming the data right and especially if you're viewing it at a higher zoom level for instance it'll just use the over views instead of downloading the full roster any other questions all right [Applause]