We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

FOSS4G Firenze 2022 - Serving earth observation data with GeoServer: COG, STAC, OpenSearch and more...

00:00

Formal Metadata

Title
FOSS4G Firenze 2022 - Serving earth observation data with GeoServer: COG, STAC, OpenSearch and more...
Title of Series
Number of Parts
351
Author
License
CC01.0Universal
Identifiers
Publisher
Release Date
Language
Production Year2022

Content Metadata

Subject Area
Genre
Abstract
Never before have we had such a rich collection of satellite imagery available to both companies and the general public. Between missions such as Landsat 8 and Sentinels and the explosion of cubesats, as well as the free availability of worldwide data from the European Copernicus program and from Drones, a veritable flood of data is made available for everyday usage. Managing, locating and displaying such a large volume of satellite images can be challenging. Join this presentation to learn how GeoServer can help with with that job, with real world examples, including: - Indexing and locating images using The OpenSearch for EO and STAC protocols. - Managing large volumes of satellite images, in an efficient and cost effective way, using Cloud Optimized GeoTIFFs. - Visualize mosaics of images, creating composite with the right set of views (filtering), in the desired stacking order (color on top, most recent on top, less cloudy on top, your choice). - Perform both small and large extractions of imagery using the WCS and WPS protocols. - Generate and view time based animations of the above mosaics, in a period of interest. - Perform band algebra operations using Jiffle. Attend this talk to get a good update on the latest GeoServer capabilities in the Earth Observation field.
Keywords
202
Thumbnail
1:16:05
226
242
SoftwareSource codeEvolutionarily stable strategyEndliche ModelltheorieRepetitionOpen setCASE <Informatik>SatelliteComputing platformServer (computing)Standard deviationOpen sourceComputer animation
SoftwareSource codeAliasingSpacetimeAttribute grammarProduct (business)Photographic mosaicOpticsHypermediaData typeDreizehnMetadataReliefAlgorithmic information theoryPeg solitaireThumbnailLink (knot theory)Object-relational mappingLanding pageWeb pageLibrary catalogContent (media)Service (economics)Virtual machineOpen setInsertion lossQuery languageOrbitComputing platformInheritance (object-oriented programming)Information securityConstraint (mathematics)Covering spacePoint cloudSerial portComputer fileElectronic mailing listCommunications protocolEndliche ModelltheorieFocus (optics)Spacetime1 (number)Open setFunction (mathematics)Scripting languageDynamical systemUniform resource locatorLink (knot theory)Search engine (computing)Medical imagingLibrary catalogInformationTemplate (C++)ImplementationService (economics)MeasurementRepresentation (politics)State observerSatelliteField of setsProduct (business)Stack (abstract data type)Latent heatAttribute grammarPoint cloudRaster graphicsWeb pageAdaptive behaviorMetadataDatabaseMultiplication signSelectivity (electronic)Server (computing)Process (computing)AreaSampling (statistics)BitLanding pageComputer animationSource code
Data typeOpticsComputing platformSerial portProduct (business)Inheritance (object-oriented programming)Source codeSoftwarePrice indexDefault (computer science)Source codeDirection (geometry)ThumbnailData managementProduct (business)Representational state transferContent (media)GeometryPoint cloudLevel (video gaming)Front and back endsStreaming mediaDatabaseComputer animation
Data storage deviceComputer filePhysical systemPlug-in (computing)Source codeSoftwareGoogolRange (statistics)Cache (computing)Product (business)Raster graphicsPoint cloudContent (media)Data structureCache (computing)Server (computing)Link (knot theory)Element (mathematics)EmailOpen setExecution unitReading (process)CASE <Informatik>Range (statistics)Local ringComputer fileData storage deviceTesselationBlogPhysical systemService (economics)CodePlug-in (computing)GoogolComputer configurationComputer animationSource code
Inclusion mapIntegerTimestampTime zoneRaster graphicsSoftwareSource codeComplete metric spacePhysical systemSubject indexingDimensional analysisPrice indexView (database)Photographic mosaicComputer-generated imageryLink (knot theory)MultiplicationComputer configurationLevel (video gaming)Representational state transferPresentation of a groupDimensional analysisRepresentation (politics)Open setUniform resource locatorDifferent (Kate Ryan album)Graph coloringProduct (business)Run time (program lifecycle phase)Domain nameSource codeDatabaseClient (computing)Musical ensembleUnified threat managementAttribute grammarQuicksortPhotographic mosaicMedical imagingSubject indexingComputer fileShape (magazine)Moment (mathematics)Link (knot theory)Table (information)Time domainDescriptive statisticsCommunications protocolCoordinate systemStack (abstract data type)Image resolutionImplementationMultiplication signServer (computing)Endliche ModelltheoriePhysical systemTime zoneComputer animationSource code
Theory of everythingPhotographic mosaicSatelliteDigital filterSoftwareSource codeView (database)Musical ensembleFunction (mathematics)Configuration spacePhotographic mosaicMedical imagingConnectivity (graph theory)CASE <Informatik>Different (Kate Ryan album)Electronic visual displayFlow separationView (database)Product (business)Graph coloringVisualization (computer graphics)Attribute grammarQuicksortComputer configurationProjective planeMusical ensembleSubject indexingMathematicsCartesian coordinate systemVector spaceCentralizer and normalizerRaster graphicsStack (abstract data type)Computer animationSource code
Photographic mosaicSlide ruleComputer animation
Source codeSoftwarePresentation of a groupDefault (computer science)Hausdorff dimensionAbstractionComputer reservations systemElectronic data interchangeComputer configurationArtistic renderingTransformation (genetics)Dimensional analysisMappingMultiplication signSubject indexingTransformation (genetics)Electronic visual displayComputer animation
Advanced Encryption StandardExact sequenceArtistic renderingComputer clusterBitAlgebraSoftwareSource codeLink (knot theory)AlgebraLevel (video gaming)Formal languageLink (knot theory)CalculationSubject indexingComputer animationSource code
Systems engineeringKeyboard shortcutExtension (kinesiology)Source codeSoftwareEmailServer (computing)ImplementationDescriptive statisticsCommunications protocolInformationComputer animationSource code
SoftwareSource codeComputing platformData structureExtension (kinesiology)Server (computing)Photographic mosaicComputer animation
Link (knot theory)Communications protocolMultiplication signProcess (computing)Computer animation
Client (computing)Process (computing)Link (knot theory)Uniform resource locatorSource codeSoftwareWindowMultiplication signUser interfacePoint (geometry)Server (computing)Process (computing)Photographic mosaicRaster graphicsMappingVector spaceComputer animation
Transformation (genetics)MetreFile formatMedical imagingMappingMultiplication signLevel (video gaming)Computer animation
UsabilityMultiplication signFrame problemRange (statistics)Time seriesServer (computing)CASE <Informatik>AdditionVideoconferencing
Source codeComputer animation
Transcript: English(auto-generated)
So my name is Andrea Aime. I work for GeoSolutions. We are one of the companies behind GeoServer. We are about open source, open standards, and today I'm going to talk about handling remote sensing imagery in GeoServer, basically. So the use case is, let's say you have a lot of satellite
imagery, grown data, and remote sensing in general, and you want to distribute it through your platform using GeoServer. How do you go about it? Well, typically the first step when you have a ton of data is finding the data that you want, so search. Initially, in GeoServer we implemented
the open search for EO, which is an ODC specification. It's an adaptation of the open search protocol geared towards space-time and satellite imagery in general. The idea is that you have a bunch of collections. The collections contain uniform products, and the products
typically point to raster files that in GeoServer we call granules. In the stack API you will call them items or actually assets. And yeah, so that's the base model.
In open search for EO, you typically do a two-step search. So the first step, the first search is to locate the collections that you are interested in, too. So for example, you say, okay, this area, this space, this time, which collections do cover this area? And you get a list of collections. You decide which ones you're interested in, too, and then you drill down into the collections
looking for the products. Say, I want, okay, this area, this time, less than 10% cloud cover. This is a sample search. So open search is a protocol based on GET request, and that URL is telling me, okay, I want to locate all the Sentinel-2 images with less than 30%
cloud coverage. And of course, I have access to a bunch of other attributes to do my search. The output typically is an RSS feed, which nowadays is a little bit outdated. I understand it. And since 2021, you can also get GeoJSON output with much of the same information.
Typically, the output has lots of links, which is common to the stack API, really. So the idea is that the RSS would not contain much information, but it would link to a metadata sheet in ISO or GML observation and measurement to describe the collection or the
product. And then, interesting enough, it would also link to OGC services to consume the data. It could also link to the data itself, like stack API does, but it's interesting to see, but in open search, the focus was still on giving you a WMS or a WCS, so an OGC protocol
that you could use to look and download the data. We implemented also stack API based on the same underlying database, so a stack API, not a catalog. So I'm talking about a dynamic
search engine that you can query much like open search for you. It's based on OGC APIs, it's fully based on JSON, open API, and so on. And we implemented the search endpoint, which allows for filtering, sorting, and field selection, cross-collection.
This is a bunch of script shots that I took from the DLR implementation of our stack API. So in GeoServer, we have templates that you can use to control both the JSON and the HTML representation of everything, like landing page, collections, items, and so on. And so DLR has made
a very good job, in my opinion, at customizing it. So here is the page showing the collections available, and I can drill down into one, get the detail about one collection, then I can drill down and get a list of the items
in that collection, and eventually get into a single item with a map, with a footprint, the thumbnail, and all the assets for download of the data, or direct usage, because they might be linking to a cloud-optimized geotiff, and then I could stream directly from
that source. We also have a back-end REST API, so that you can go and manage the collections, the products, the granules, and so on, so that you can automate the management of the contents
of the database. Okay, say that I found what I want. I know which products I want. So now, how do I access that raster data? Well, as I said, stack and open search, they both have links, typically to services, but also to the unit elements that make up the product, so the assets.
GeoServer-wise, they might be located on the local file system, which is kind of the most common use case, but also the less interesting one, or they could be put on an HTTP server, on an S3 bucket, or Azure blob, or any cloud blob storage, and we can read directly from
those blob stores using the COG plugin, the cloud-optimized reader, which leverages the code structure. In GeoServer, we also have an older reader that could read literally any geotiff, no matter whether it was formatted as a COG or not. It's less efficient, but it's more general.
The COG plugin is smart enough to get the header of the file, and then just do HTTP range reads to just get the tiles that it needs without having to read anything else. It currently works against S3, HTTPS, Google storage. We are implementing it on Azure blob these
days, like it's going to be available in weeks. We might get local caching of the blob contents, which would be interesting for speed. If you're interested, please contact me. Of course, more blob storage options. No. Okay, I can get to the raster data.
How do I put it together? Typically, with an image mosaic. Typically, I've located a bunch of products, not just one, and I want to see them. We use the GeoServer Mosaikin machinery to
go and locate the data. The Mosaikin machinery is powerful enough that I can fetch and mosaic together data in different coordinate reference systems, different color models, different resolutions, which is, by the way, very important for, say, Sentinel-2, where each product is in a different UTM zone, and different bands have different resolutions,
and so on. This kind of freedom is actually very important to get things going. The Mosaikin index, typically, is stored in a database, like PostGIS, Oracle SQL Server, but it could also be located in a shapefile or other sources. I'm going to get there in a moment.
Well, it's typically looking more or less like a table, where I have the location of the image, its footprint, but also all the attributes that I would like to use for filtering and sorting and stacking the images. The time, the elevation, the runtime of the weather model, and so on. As I was saying, we have dimensions.
Dimensions are then recognized by OGC protocols. If I look at the WMS capabilities document or the elevation domain and so on, that the clients can then use to filter down the data that they want.
Interesting enough, we can use the stack slash open search database as an index for image Mosaik. If I ever implemented stack or open search in my system, there is a quick REST API that I can use to say, please make me a layer out of this collection. It literally
pipes directly into that database. What I'm working on, literally these days, I'm sort of halfway in the implementation, is the idea of having GeoServer connected to an external stack API, treat it as an index of sorts, and then being able to display on the map
the footprints, filter them, and then eventually flip to raster representation if the stack API is pointing to cogs, to cogs that I can access. A bunch of links, they are there for you to follow
when you get a hold of the presentation slides, if you are interested. With image Mosaik, we can do a bunch of interesting stuff. When you are Mosaik-ing together several products, you probably want to filter, and Mosaik-ing allows you to filter on any attributes in the index, but also decide the stacking. I want the color images on top,
or I want the most recent images on top, or less cloudy images on top. You can perform sorting, and if I'm talking to a stack API, hopefully it's implementing the sorting option, and I will literally flip the sort onto the stack API, get the items that I want, and
Mosaik-ing the imagery accordingly. Another interesting thing is coverage views, which is this idea that I might have several raster layers that represent maybe different bands or different components of a phenomenon, and stick them back together in a single raster that I can then use for visualization. This use case is showing taking two bands, which are U and
V, the projection of the wind vector on the two axes, and putting them back together so that I can do math on them and display wind barbs. Another use case is taking central image imagery and do false color imagery, or I'm going to get to it. I thought I had the NDVI slide,
but it's going to get there. Once I am ready to use the Mosaik, I'm ready to visualize the data. Visualize through WMS or to OGC API maps. WMS is going to export all the dimensions that
I have in the capabilities document. I will be able to quickly filter on time and elevation and custom dimensions through the URL. I will be able to do custom filtering using SQL filter or sorting and so on. All that I do on the WMS is going to be packed down into the original
index, which might be the stack API, and then I'm going to talk to the stack API and say, this filter, this sorting. We can do a bunch of rendering transformations, which is the idea of taking data and processing it quickly to get a different display. So contouring,
wind barbs, currents, but also NDVI on the fly. So we have our little map algebra language that we can stick into the styles to perform simple index calculations such as NDVI, NDWI,
and so on and so on. And a bunch of other links that you can use. Then if you are satisfied with the data and you actually want to download it, you can use WCS, which is the OGC protocol to do that. The WCS 2.0 implementation into
your server is complete. The protocol is same, simple to use. And well, in the description of the coverage, you get all the spatial and temporal information about the coverage. And we added a few vendor extensions in GeoServer so that we can specify the same
filtering and the same sorting that we applied in WMS so that I saw something and then I can download that particular mosaic with those products inside and get a geotiff out of it with the same structure. However, WCS has some limitations in that the requests are synchronous. So I'm
sending the request, waiting on the HP link, and maybe I'm waiting half an hour, and I'm not going to wait that long because something is going to time out my request. So what do I do about large downloads? I can use WPS. WPS is the OGC protocol that has
asynchronous request support built in. So we implemented in GeoServer a process that's called WPS download, which is designed to allow large downloads of both vector data and raster data. So I can hit a very large image mosaic and wait whatever time is needed and then sometime later
fetch 25 gigabytes worth of geotiff out of GeoServer without the timeouts. So of course, you're going to have this user interface telling the user, okay, I'm waiting for the
download to come back later. And at some point, it's going to notify that the download is available. We can also do large rendered maps. So maybe what you want to do is actually to to have a printable geotiff like 20,000 by 40,000 NDVI properly color the image,
which is also something that you wouldn't be able to do with get a map with WMS for the same reason. It's synchronous. So yes, I'm almost done. And so with WPS download, we can say, yeah, I want to download this very large map, and it's going to take whatever time I'm going to wait.
In addition to that, we can do long animations. So say that I have, like in this case, a Meteosat time series, and I wanted to make an MP4, a video out of it. With WPS download, I can also instruct the server to work on a certain layer and a certain amount of time range, build the
frames, build my animation, and after some time, I can download my MP4 with the animation. And with that said, I'm done.