The State of Cloud-Native Geospatial
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 351 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/69202 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Year | 2022 |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
00:00
Query languageForestAverageTotal S.A.NumberBit rateElement (mathematics)ForestBuildingOpen sourceRight angleNeighbourhood (graph theory)Computer animation
00:59
Google EarthQuicksortView (database)Point cloudComputer animation
01:26
Point cloudDirected setScalabilityOnline-PublizierenProcess (computing)Visualization (computer graphics)Audio file formatService (economics)DemosceneVisualization (computer graphics)EstimatorAnalytic setReal numberReplication (computing)Scaling (geometry)Type theoryTerm (mathematics)DivisorCodeClient (computing)Scripting languageLaptopNumberScalabilityProcess (computing)Point cloudInternet service providerIntegrated development environmentObservational studyComputer animation
03:11
Open sourceStandard deviationService (economics)Audio file formatMetadataExploit (computer security)Standard deviationOpen sourceService (economics)Stack (abstract data type)BuildingComponent-based software engineeringInternet service providerRight angleQuicksortSet (mathematics)Audio file formatGeometryComputer animation
04:22
Element (mathematics)SpacetimeExistenceSatelliteInformationIntegrated development environmentDecision theoryOpticsImage resolutionElectric currentSurfaceRule of inferenceMobile appSoftwareWebsiteComputer-generated imagerySlide ruleOpen setWindows RegistryArtificial intelligenceFacebookDigital signalDecision tree learningComputer programRepository (publishing)Variety (linguistics)Stack (abstract data type)Point cloudRepository (publishing)Open setAudio file formatComputer animation
04:52
Data storage deviceComputer fileTesselationOrder (biology)Set (mathematics)Computer animation
05:15
GeometryMultiplication signFile archiverRight angleLink (knot theory)TesselationComputer animation
05:56
SineMetadataGeometrySharewareRight angleWeb portalComputer animation
06:38
Perspective (visual)Scale (map)Process (computing)Raster graphicsPerturbation theoryOpen setSpacetimeSatelliteSoftwareComputing platformLibrary catalogFluid staticsStandard deviationGoogolRepository (publishing)Product (business)HexagonMetadataClient (computing)ImplementationBit rateVideoconferencingCore dumpSuite (music)Interface (computing)Element (mathematics)Web crawlerInternationalization and localizationGeometryStack (abstract data type)Open sourceMathematical analysisTemporal logicPoint cloudField (computer science)Extension (kinesiology)RootLocal GroupLatent heatLink (knot theory)Library catalogFluid staticsMetadataRight angleInternetworkingUniform resource locatorStack (abstract data type)Point (geometry)Range (statistics)DemosceneGreatest elementComputer fileCore dumpExtension (kinesiology)Stability theoryLatent heatSelf-organizationGroup actionDatabaseLevel (video gaming)GradientConsistencyTesselationTemporal logicService (economics)Subject indexingStandard deviationMultiplication signStationary stateQuery languageComputer animationProgram flowchart
09:28
Software developerLatent heatRevision controlStack (abstract data type)Computer animation
10:07
Core dumpLink (knot theory)Field (computer science)Object (grammar)String (computer science)GeometryData typeFinitary relationExtension (kinesiology)Sheaf (mathematics)Data dictionaryRevision controlUniqueness quantificationInternet service providerDefault (computer science)MetadataUniform resource locatorElectronic mailing listRaster graphicsPixelMusical ensembleMetreSatelliteFrequencyImage resolutionMaxima and minimaProduct (business)AreaStandard deviationStatisticsOpen sourceIntegerPolarization (waves)CodeDimensional analysisComputer reservations systemNumberDirection (geometry)Range (statistics)Affine spaceTransformation (genetics)CoefficientAxonometric projectionPhysical systemCoordinate systemPoint cloudEstimationOpticsSample (statistics)Asynchronous Transfer ModeWaveAzimuthAverageAngular resolutionType theoryProcess (computing)Service (economics)Element (mathematics)Vector spaceOpen setLibrary catalogLanding pageInformationLocal GroupPredicate (grammar)CodeMetadataLink (knot theory)1 (number)Extension (kinesiology)RootLibrary catalogSoftwareStack (abstract data type)Core dumpQuicksortLatent heatComplex (psychology)Raster graphicsAdditionProjective planeDigital object identifierMusical ensembleCategory of beingGeometryRegular graphInheritance (object-oriented programming)Multiplication signRight angleSelf-organizationElectronic mailing listComputer fileComputer animation
12:14
Computer fileAudio file formatRaster graphicsPoint (geometry)Vector spaceData typeBitFile formatRight anglePoint cloudPoint (geometry)GeometryPixelCASE <Informatik>TesselationDefault (computer science)Set (mathematics)Computer fileAudio file formatComputer animation
13:18
Smith chartFile formatDiagramSet (mathematics)TesselationPoint cloudComputer fileData storage deviceCASE <Informatik>Centralizer and normalizerComputer animation
13:57
Audio file formatComputer fileStructural loadTime zoneComputer-generated imageryPlane (geometry)Function (mathematics)outputAxonometric projectionQuery languagePlot (narrative)Library catalogLetterpress printingImage resolutionInstallable File SystemEquivalence relationOpen setCubeDigital signalDimensional analysisService (economics)Personal digital assistantStandard deviationStorage area networkBootstrap aggregatingLaptopLibrary (computing)CASE <Informatik>TesselationQuicksortPoint cloudAudio file formatKey (cryptography)Open sourceLibrary catalogComputer fileDifferent (Kate Ryan album)Process (computing)ScalabilityPower (physics)Mathematical optimizationRight angleCubeData storage deviceBitOpen setReading (process)Standard deviationProjective planeNeuroinformatikMultiplication signGeometrySoftwareLatent heatServer (computing)Stack (abstract data type)ImplementationSoftware developerCodeWeightMetadataDatabaseComputer animation
18:33
PixelDigital signalQuantum stateSpacetimeComputerGoogolMultiplication signConsistencySlide ruleSet (mathematics)NeuroinformatikLibrary catalogPoint (geometry)Revision controlOpen sourceElement (mathematics)Subject indexingInformation privacyStack (abstract data type)Computer animation
19:41
Solid geometryQuantum statePhysical systemMedical imagingError messageOrientation (vector space)Computer-generated imageryOrbitSurfaceControl flowSubsetConsistencyFraction (mathematics)PixelScale (map)Image resolutionEntire functionUsabilityMathematical analysisSequenceSimilarity (geometry)MetadataGlobale BeleuchtungSphereRevision controlAverageTemporal logicInternet forumStandard deviationComputer animation
20:03
Library (computing)Utility softwareMetadataExecution unitField (computer science)Function (mathematics)Revision controlLibrary catalogEndliche ModelltheorieInformationRepository (publishing)ImplementationClient (computing)Router (computing)Element (mathematics)Common Language InfrastructureLocal ringWeb browserCore dumpContent (media)Table (information)Covering spaceInterface (computing)Line (geometry)Electronic visual displayOpen sourceSoftwareProjective planeOpen setClient (computing)Repository (publishing)Slide ruleRow (database)Utility softwareComputer animation
20:35
Computer fileRevision controlHard disk driveComputer fileTerm (mathematics)Multiplication signMedical imagingInformationPoint cloudRight angleDirectory serviceAlgorithmSocial classSource codeComputer animation
21:36
MetadataAlgorithmMetadataRevision controlUniform resource locatorCASE <Informatik>Computer animation
22:24
RootElement (mathematics)Inheritance (object-oriented programming)Core dumpProcess (computing)SoftwareComputing platformOpen sourceAlgorithmFunction (mathematics)Row (database)TrailInformationPrice indexRevision controlPoint cloudMetadataStack (abstract data type)Open sourceInversion (music)Computer animation
22:48
ArchitectureWavePower (physics)Service (economics)BuildingPoint cloudComputer animation
23:27
Computer animation
Transcript: English(auto-generated)
00:02
Hello, everybody. My name is Matt Hansen. I work for Element 84. I'm the geospatial engineering lead there. And today, I'm going to talk about cloud-native geospatial. So first, I want to talk about this concept of a queryable Earth. So these are actually often the questions
00:20
it is that people want to know about our world. How many ships are in this port? How many buildings were built in my neighborhood? How many forests have been, how many acres of forest have been deforested? These are the questions that we ultimately really it is that we're after. Notice what is missing from all of these questions.
00:42
What's missing is the actual data source, right? We're not saying, oh, with MODIS data, we're going to do this, right? That's ultimately, that's not really what it is that we're interested in. That's what we have to deal with as data scientists. We have to deal with MODIS or Landsat. But that's really not the end goal.
01:01
So keep that in mind. So what is cloud-native geospatial? I'm going to be taking sort of a big picture view here. So it's a little different than the previous talks. There'll be some overlap with that. And I just kind of want to talk about big picture. What is it that we're talking about when we talk about cloud-native geospatial? And it's Google Earth Engine.
01:21
It really is what it is. You've all probably heard of Google Earth Engine. It's about having complete workflows in the cloud, from data discovery through the processing, the analytics, and the visualization. All of that happens through a lightweight client that the user is using. They have direct access to the data, but a lot of it
01:42
is abstracted and it's transparent to the user. And Google Earth Engine is great. You type a script, and you don't worry about individual scenes. You're not worrying about the data. You just can create a mosaic over the entire world if you want to. And so this is what cloud-native geospatial is.
02:00
You have direct access to the data. And this promotes scalability, of course, and reproducibility, data provenance, which I'll talk about, and online publishing. I like this concept of executable papers. I don't know if any of you have heard this term. I didn't make it up. I don't know where it originally came from. But it's this idea that rather than just publishing
02:22
your paper in a journal, you actually publish something where there's a button, and somebody can hit that button, and they can open up your notebook, your study, in their own environment, like in Google Cloud or in AWS or whatever it may be, and they can run it. And they would pay for that processing,
02:40
but they can replicate it, because there's a real reproducibility problem in science. Some estimates say that more than half of the papers actually can't be reproduced. And this is due to a large number of factors. The code is often not available. The data is hard to get to. You have to download individual scenes.
03:01
And there's no way that you could take a really large scale analytics paper and actually be able to reproduce that. So Cloud Native Geospatial I say is really it's a collection of standards and technologies for this easy access discovery and exploitation
03:21
of the geospatial data in the cloud, of course. Except the problem with Google Earth Engine is it's not open source. It's not based on any sort of standards. You can't deploy it yourself. You can't really easily integrate it with commercial providers, and you can't really extend it. You can't build services on top of it.
03:43
Now, I'm sure somebody from GE would come to me and say, no, actually, we can build services. But that's not really the primary goal. It's about using their service. So Cloud Native Geo is really about how do we take what Google Earth Engine did and turn that
04:00
into an open source set of tools that now everybody can use? And there's a few different components of that. There's stack. There's cloud optimized data formats. And then there's this other thing, ARD, which I'm not going to talk about today. I have a lightning talk tomorrow. It's sort of ill-defined. It doesn't really mean anything. If you want to know more about what it doesn't mean,
04:21
then you can come to my talk tomorrow afternoon. So let's talk about stack. And people have been going in and out. So you might not have been at the previous two talks. But again, there'll be some overlap here. So let's take a look at AWS Open Data. So AWS has these great repositories of data online that are publicly available.
04:42
And you can go to it, and you can access the data. Sometimes it's a request or pays bucket. Sometimes it's not. But it's great because you have this data in cloud optimized formats. The problem is that in order to find the data that you need, you kind of have to construct what that file name is.
05:01
This is how you still have to do it with a lot of data sets. So you figure out the tile that you want, and then you can construct what the file name is in this blob storage on S3. And then you can go, and you can look and see what data is there. Well, that's just really not ideal at all because a lot of data is broken up into tiles,
05:21
and it might not completely be covered with data. And so my AOI is over here on the right-hand side. The data only covers the left-hand side. I spend all this time downloading the data, accessing it, and I have nothing at all. I've really just wasted my time. The NASA archive, as just one example of the explosion of data, is estimated to be
05:42
over 250 petabytes in the next several years. The NISAR mission, which is to be launched next year, is gonna have a downlink of 85 terabytes every day. So that's just an explosion of data. So if your metadata isn't indexed in something that's searchable,
06:02
then that data might as well not exist. People are not gonna be able to access it. And so a lot of metadata is indexed, and it's available via these GeoPortals. And you can go to them, and you could download some data. This is like, really, any data scientist is like, yeah, this is the worst way to actually get data.
06:22
It's great as a demo. It's great maybe to get some funding. But this is really terrible. Most of these GeoPortals, of course, will be backed by an API, which is great. But there traditionally has been, before Stack, there really wasn't any real consistency between all that.
06:41
So back at the state of the map in Boulder in 2017 is really where Stack started. It was called the imagery metadata spec there. And there was a bunch of organizations that were involved with that. And the requirements really were on, how do we get metadata and allow users
07:01
to do search and discovery in an easy way? We wanna leverage other standards. We don't wanna reinvent the wheel. We don't wanna replace existing metadata standards. We're not looking to replace some IEEE standard. And we want the metadata to be crawlable. So this is a central idea behind the internet, right?
07:21
Like, the internet's crawlable. And so, with Stack, the idea is that you can have static metadata, and you can crawl it and index it. So it doesn't need to necessarily be in a database. It can be a bunch of static files. Therefore, you don't need to stand up any service. And then if you do want to search it, it can be crawled and then put into a database.
07:42
And we also wanted a simple extensible core. So extensions, there's pros and cons to that approach, but the core of Stack, we wanted to be really simple so that it would be stable. I'll skip the timeline here.
08:01
Okay, so what is Stack? Really briefly, there's catalogs, there's collections, there's items, and then down at the bottom, there's assets. It's all in JSON. All of these entities are in JSON. Again, it's crawlable. There's an API for querying. Catalogs are really just containers that point to other things. Collections are a way to group typically scenes.
08:24
So like Landsat, that would be a collection. Sentinel-2, that would be a collection. The items are the actual individual scenes. So it's a collection of data files. It's more than one asset typically that's for a specific location at a specific date and time. So it's a when and a where,
08:42
although if you have a mosaic, it could be a range of times, but the same idea applies. So here's what a larger catalog might look like. You'll see that there's some grayed out catalogs. Those are just sub-catalogs. Those are for organizational purposes. You could organize things this way.
09:02
In an API, you really don't have this because you really don't have these sub-catalogs because you can search. But in a static catalog, you might partition the data in various ways, such as when I showed before, there was the discovery by file name. Things are separated by the tile location and then the year and then the month and so on. And then down at the bottom, we have these assets.
09:21
And that's why it's called the spatial temporal asset catalog is because ultimately it's the assets that we're after. That's the actual data that we want. So the stack specifications, they describe the geospatial data, that's the stack spec, and then how to search that data, that's the stack API spec.
09:40
It was about over a year ago now that the version one was released. Stack API spec is still not on a 1.0 version and it's because we're tied to, it's a longer story. But we've been tying ourselves to OGC API. It's really built on the OGC API features.
10:01
And so that's been under development now. That has a 1.0 version. So we should be seeing an API spec version come out shortly. So like I said, there's a simple core spec. This is GeoJSON, should be familiar to most people here. There's the geometry properties. You'll see the links and assets are something
10:21
that might not be familiar to regular GeoJSON. And that's really kind of the crux of it right there. The assets is the actual data files. And links is, to really serve two purposes. You have hierarchical links. That's how we go up and down and are able to crawl catalogs. So an item will point back to its parent, its collection and its root catalog and so on.
10:44
But it's also, we encourage judicious use of links to other things. It could be to documentation or to a DOI or code, additional code to use it. So the links are critical.
11:01
So it's a simple core spec. The complexity comes in with stack extensions. So there's a whole list of stack extensions. There's a GitHub organization, stackextensions.github.io. And it's really, it's open to anybody to add extensions. The idea is that the marketplace will eventually win, is kind of the idea, right?
11:21
Like stack extensions, anybody can implement extension. Anybody can put it in an item. It'll still gonna be compatible with software that's just looking at the core stack metadata. And over time, we'll see, oh, this extension is really useful and this extension is useful. And that's where the best practices come in
11:40
that Pete talked about earlier, is what are the sorts of extensions that we should be putting into stack? And so some of the notable ones are the projection and raster extensions. Those are important for the ODC stack software. And then we have some sensor specific extensions like for SAR or for electro optical to define spectral bands.
12:05
Okay, and then I said the stack API spec is based on the OGC API features with just a couple search endpoints to do cross collection searches, what that's for. Okay, cloud optimized file formats. Gonna switch gears a little bit. You've heard a lot about this, right? COGS and SAR, this has been talked about.
12:23
Cloud optimized point clouds, geo parquet. And I'm not gonna talk about all these data formats. I just wanna tell you about like what is it that makes something cloud optimized to begin with? So in a traditional file, the default in GDAL would be you create something and there's these striped chunks.
12:41
So now the important thing to know here is that each one of those chunks is compressed individually, okay? So you can't read just a piece of a chunk. You have to read the whole chunk and then you decompress it and then you could pull out just the pixels that you need. And so if that's my AOI over there, then I need to read both those chunks.
13:01
I need to decompress them and then I can pull out the pixels that I need. So cloud optimized really just means that we have tiles that are more suited to the kinds of use cases that we want. And so if this is my AOI, I only have to read this one tile and that's great. It's a lot less data that I have to read. So a cloud native data set
13:21
is really one with small addressable chunks. That's it. It's as simple as that. Like people will talk about overviews, but that's not strictly necessary. It's really about this central concept. It's a small addressable chunk. Now those chunks could be via internal tiles like in the diagram that I showed, or in the case of ZAR,
13:41
they're actually individual files in blob storage. So people might call it an exploded file format where all of our tiles are actually individual blobs in blob storage. And so I can address each one of those and I only have to download that. And that's it. The cool thing about COGS
14:02
and the reason why it took off so much is because TIFF is a really common format. And so really you can take old formats as NetCDF and HDF and there are now ways, there's a library called Kerchunk to actually generate what all of those offsets are
14:21
to all of the internal tiles and store that as a sidecar file. So now you can read NetCDF and HDF as if it was cloud optimized. And this is really cool because we can take legacy formats and we don't necessarily have to convert them and we can make them cloud optimized. Matthias mentioned before the Kotar.
14:41
This was actually a joke at this cloud outreach day that we had where somebody said, oh cloud optimized tar file and turns out it actually existed. Somebody had made it. And we were just joking. But you look into it and of course it's a cool idea. It's like it made sense for that use case is you have tar files and you just have a sidecar file that says where in the file
15:01
that you need to look for certain files and then certain chunks within those files. So it's not just about taking in one piece, right? Like I showed an AOI and it's like, oh I can read this one little piece of data. But the real beauty, the scalability piece here is that if I use something like Dask, right?
15:22
I'm using X-ray in my Python notebook and I have a bunch of Dask workers. Each one of those workers can now go and read a piece of the cog. And so this is really where the power comes in. It's not just for those people who just don't want to download a single file
15:40
and they just want an AOI. It's the fact that this huge file can be broken up into lots of little different AOIs and they can all be performed simultaneously. So ODC Stack, there's the Open Data Cube project from Geoscience Australia has been pushing forward for years now.
16:01
And this is a recent project that was funded by the Planetary Computer to essentially take the Open Data Cube code and rather than requiring a Postgres database, it can now use stack metadata. So you could point this at a stack API, you could do a search, and now you can open this up as an X-ray. It's a great tool, I suggest you use it.
16:22
All right, I just want to switch gears a little bit and talk about standards. So because these are, right here we're talking, these are standards, right? Like stack's a standard, sorta, is really a specification. COG, that's a standard, but not really, it's a specification. What makes them standards? Like they're not backed by anything official.
16:41
So some of you might have seen this XKCD comic, which is a great comic, which I kinda disagree with a little bit because there's competition here, right? Like, okay, you have 15 standards, it doesn't mean that they're all equal.
17:00
So I drew this yesterday quickly just to get the idea across, but for successful standards there's, the keys are adoption and having a software ecosystem. And they feed each other, right? To have a successful standard, to take a specification and turn it into a standard, you need adoption, you want lots of people to be using it.
17:22
That's the key, that's ultimately what you want. So how do you get people to use it? You don't just give them a spec and say you should use this. That's the worst thing. This is where a lot of standards have failed. You need to build an ecosystem around this. And so you see outreach and open source development over here. This is how we sort of bootstrap this process.
17:43
So to increase adoption, we do a lot of outreach. I'm always out there talking, lots of other people are out there talking and giving talks and doing stuff online about all the benefits here of Cloud Native Geo. And we have open source development,
18:01
we fund this or meet people do it, champions do this in their spare time to add to the software ecosystem, to add to the implementations. When users go out there and they see this big software ecosystem of tooling that they can use, this is why stack and cogs have been successful
18:20
because you can use it right away. Like early on there was servers and there was data catalogs of Sentinel and Landsat data that you could use and people were benefiting from early on. So adoption, I know I'm running out of time so there's probably like 40 slides left in this deck,
18:41
but that's okay. A lot of this has been touched on. So there's a lot of users, there's a lot of public stack APIs out now that have been mentioned. I'll just point out the last one here, stack index. That's a great resource. You can go there and you can learn about all these public catalogs.
19:01
The earth search one is one that we maintain at Element 84 for the Sentinel-2 data and there's an official USGS Landsat data. But the Microsoft planetary computer is a great example because they've really taken a top down approach where they're ingesting all this data and making all of the data available in a stack API
19:22
in a really consistent way. And there's nothing quite like it. Like I hoped that we can see something like this on AWS for all their public data sets at some point, but right now the planetary computer is kind of the, it's really a great example of an open source version of Google Earth Engine.
19:41
So let me, oh, and this is a cool thing too. A couple of years ago, somebody from USGS astrogeology reached out and they're like, oh yeah, we're using stack for Jupiter and Mars. And we're like, really? Like, you know, you could do that? And they use QGIS and stuff. I had no idea. It was really cool to see that these standards can be used
20:01
for other planets as well. So Pete talked about the software ecosystem. There's just a couple of slides here. Stack utils is kind of, it's not the end all and be all. Like people, anybody can make software, but this is an open GitHub org that people can add to and contribute to and you can move your project there if you want it.
20:22
And there's a whole lot of really, really great repositories there. PyStack client is one of the main things that users would be interested in because it's for searching APIs. And Pete had talked about stack tools. You can go and watch the recording of his. So lastly, I want to talk about this concept of cloud native geospatial workflows.
20:43
I really want people to stop thinking in terms of files. So this here is your typical, I've been using this image for a long time. This is like your typical image of like a scientist's work directory. It's like they vomited on their hard drive. And if something happens, right, they leave
21:03
and now somebody comes in and they're like, okay, well, which version was published here? Okay, is it final version class? Is it final version final because it's got the word final in there twice? Like which one is it? And you won't know because there's no provenance information.
21:22
There's nothing there. So because the way that they used to do things, would do things is, okay, I have an algorithm. It's going to read in a file and it's going to generate another file. And you're kind of relying on them knowing what produced what. So start thinking of things rather than writing algorithms that work on data,
21:43
write algorithms that work on metadata. So the metadata, in this case, stack, right? You have stack item or one or more stack items and that goes into your function. And that contains a complete characterization of the data, including the location of that data and how to access it.
22:01
And the beauty here is that your workflow can do whatever it wants. It can fetch the data if it needs to. Maybe it doesn't need to. Maybe it only needs to fetch certain pieces of the data. Maybe it's adding an asset. Maybe it's adding to the metadata. And then on the other end, it spits out more metadata.
22:21
That's a modified version or maybe a new item altogether. And this way you can really easily track data provenance because of the workflow. This algorithm can add in the source stack item. It can add in lineage information. It can add in version information really easy and then you output it and you have that metadata record.
22:42
It's lightweight. And maybe your algorithm has pushed data up to the cloud. And you have an indication of where that's located. So I'm gonna leave you with this. Cloud Native Geospatial is really the foundational architecture for this concept of a queryable earth. If you're not using Cloud Native Geospatial,
23:01
if you're building services, you should really rethink what you're doing. Now this isn't to say that you shouldn't build services, but if you build services, build them on top of a Cloud Native Geospatial architecture where the service is directly accessing the data. This has a lot of power to it. And it's the wave of the future, I would say.
23:23
So thank you.