FOSS4G Firenze 2022 - Serving earth observation data with GeoServer: COG, STAC, OpenSearch and more...
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 351 | |
| Author | ||
| License | CC01.0Universal | |
| Identifiers | 10.5446/69218 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Year | 2022 |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
FOSS4G Firenze 202227 / 351
1
7
13
22
25
31
33
36
39
41
43
44
46
52
53
55
58
59
60
76
80
93
98
104
108
127
128
133
135
141
142
143
150
151
168
173
176
178
190
196
200
201
202
204
211
219
225
226
236
242
251
258
263
270
284
285
292
00:00
SoftwareSource codeEvolutionarily stable strategyEndliche ModelltheorieRepetitionOpen setCASE <Informatik>SatelliteComputing platformServer (computing)Standard deviationOpen sourceComputer animation
00:35
SoftwareSource codeAliasingSpacetimeAttribute grammarProduct (business)Photographic mosaicOpticsHypermediaData typeDreizehnMetadataReliefAlgorithmic information theoryPeg solitaireThumbnailLink (knot theory)Object-relational mappingLanding pageWeb pageLibrary catalogContent (media)Service (economics)Virtual machineOpen setInsertion lossQuery languageOrbitComputing platformInheritance (object-oriented programming)Information securityConstraint (mathematics)Covering spacePoint cloudSerial portComputer fileElectronic mailing listCommunications protocolEndliche ModelltheorieFocus (optics)Spacetime1 (number)Open setFunction (mathematics)Scripting languageDynamical systemUniform resource locatorLink (knot theory)Search engine (computing)Medical imagingLibrary catalogInformationTemplate (C++)ImplementationService (economics)MeasurementRepresentation (politics)State observerSatelliteField of setsProduct (business)Stack (abstract data type)Latent heatAttribute grammarPoint cloudRaster graphicsWeb pageAdaptive behaviorMetadataDatabaseMultiplication signSelectivity (electronic)Server (computing)Process (computing)AreaSampling (statistics)BitLanding pageComputer animationSource code
04:27
Data typeOpticsComputing platformSerial portProduct (business)Inheritance (object-oriented programming)Source codeSoftwarePrice indexDefault (computer science)Source codeDirection (geometry)ThumbnailData managementProduct (business)Representational state transferContent (media)GeometryPoint cloudLevel (video gaming)Front and back endsStreaming mediaDatabaseComputer animation
05:03
Data storage deviceComputer filePhysical systemPlug-in (computing)Source codeSoftwareGoogolRange (statistics)Cache (computing)Product (business)Raster graphicsPoint cloudContent (media)Data structureCache (computing)Server (computing)Link (knot theory)Element (mathematics)EmailOpen setExecution unitReading (process)CASE <Informatik>Range (statistics)Local ringComputer fileData storage deviceTesselationBlogPhysical systemService (economics)CodePlug-in (computing)GoogolComputer configurationComputer animationSource code
06:42
Inclusion mapIntegerTimestampTime zoneRaster graphicsSoftwareSource codeComplete metric spacePhysical systemSubject indexingDimensional analysisPrice indexView (database)Photographic mosaicComputer-generated imageryLink (knot theory)MultiplicationComputer configurationLevel (video gaming)Representational state transferPresentation of a groupDimensional analysisRepresentation (politics)Open setUniform resource locatorDifferent (Kate Ryan album)Graph coloringProduct (business)Run time (program lifecycle phase)Domain nameSource codeDatabaseClient (computing)Musical ensembleUnified threat managementAttribute grammarQuicksortPhotographic mosaicMedical imagingSubject indexingComputer fileShape (magazine)Moment (mathematics)Link (knot theory)Table (information)Time domainDescriptive statisticsCommunications protocolCoordinate systemStack (abstract data type)Image resolutionImplementationMultiplication signServer (computing)Endliche ModelltheoriePhysical systemTime zoneComputer animationSource code
09:22
Theory of everythingPhotographic mosaicSatelliteDigital filterSoftwareSource codeView (database)Musical ensembleFunction (mathematics)Configuration spacePhotographic mosaicMedical imagingConnectivity (graph theory)CASE <Informatik>Different (Kate Ryan album)Electronic visual displayFlow separationView (database)Product (business)Graph coloringVisualization (computer graphics)Attribute grammarQuicksortComputer configurationProjective planeMusical ensembleSubject indexingMathematicsCartesian coordinate systemVector spaceCentralizer and normalizerRaster graphicsStack (abstract data type)Computer animationSource code
10:41
Photographic mosaicSlide ruleComputer animation
10:55
Source codeSoftwarePresentation of a groupDefault (computer science)Hausdorff dimensionAbstractionComputer reservations systemElectronic data interchangeComputer configurationArtistic renderingTransformation (genetics)Dimensional analysisMappingMultiplication signSubject indexingTransformation (genetics)Electronic visual displayComputer animation
11:45
Advanced Encryption StandardExact sequenceArtistic renderingComputer clusterBitAlgebraSoftwareSource codeLink (knot theory)AlgebraLevel (video gaming)Formal languageLink (knot theory)CalculationSubject indexingComputer animationSource code
12:09
Systems engineeringKeyboard shortcutExtension (kinesiology)Source codeSoftwareEmailServer (computing)ImplementationDescriptive statisticsCommunications protocolInformationComputer animationSource code
12:38
SoftwareSource codeComputing platformData structureExtension (kinesiology)Server (computing)Photographic mosaicComputer animation
12:59
Link (knot theory)Communications protocolMultiplication signProcess (computing)Computer animation
13:25
Client (computing)Process (computing)Link (knot theory)Uniform resource locatorSource codeSoftwareWindowMultiplication signUser interfacePoint (geometry)Server (computing)Process (computing)Photographic mosaicRaster graphicsMappingVector spaceComputer animation
14:05
Transformation (genetics)MetreFile formatMedical imagingMappingMultiplication signLevel (video gaming)Computer animation
14:39
UsabilityMultiplication signFrame problemRange (statistics)Time seriesServer (computing)CASE <Informatik>AdditionVideoconferencing
15:11
Source codeComputer animation
Transcript: English(auto-generated)
00:00
So my name is Andrea Aime. I work for GeoSolutions. We are one of the companies behind GeoServer. We are about open source, open standards, and today I'm going to talk about handling remote sensing imagery in GeoServer, basically. So the use case is, let's say you have a lot of satellite
00:24
imagery, grown data, and remote sensing in general, and you want to distribute it through your platform using GeoServer. How do you go about it? Well, typically the first step when you have a ton of data is finding the data that you want, so search. Initially, in GeoServer we implemented
00:44
the open search for EO, which is an ODC specification. It's an adaptation of the open search protocol geared towards space-time and satellite imagery in general. The idea is that you have a bunch of collections. The collections contain uniform products, and the products
01:04
typically point to raster files that in GeoServer we call granules. In the stack API you will call them items or actually assets. And yeah, so that's the base model.
01:20
In open search for EO, you typically do a two-step search. So the first step, the first search is to locate the collections that you are interested in, too. So for example, you say, okay, this area, this space, this time, which collections do cover this area? And you get a list of collections. You decide which ones you're interested in, too, and then you drill down into the collections
01:45
looking for the products. Say, I want, okay, this area, this time, less than 10% cloud cover. This is a sample search. So open search is a protocol based on GET request, and that URL is telling me, okay, I want to locate all the Sentinel-2 images with less than 30%
02:04
cloud coverage. And of course, I have access to a bunch of other attributes to do my search. The output typically is an RSS feed, which nowadays is a little bit outdated. I understand it. And since 2021, you can also get GeoJSON output with much of the same information.
02:26
Typically, the output has lots of links, which is common to the stack API, really. So the idea is that the RSS would not contain much information, but it would link to a metadata sheet in ISO or GML observation and measurement to describe the collection or the
02:45
product. And then, interesting enough, it would also link to OGC services to consume the data. It could also link to the data itself, like stack API does, but it's interesting to see, but in open search, the focus was still on giving you a WMS or a WCS, so an OGC protocol
03:06
that you could use to look and download the data. We implemented also stack API based on the same underlying database, so a stack API, not a catalog. So I'm talking about a dynamic
03:24
search engine that you can query much like open search for you. It's based on OGC APIs, it's fully based on JSON, open API, and so on. And we implemented the search endpoint, which allows for filtering, sorting, and field selection, cross-collection.
03:44
This is a bunch of script shots that I took from the DLR implementation of our stack API. So in GeoServer, we have templates that you can use to control both the JSON and the HTML representation of everything, like landing page, collections, items, and so on. And so DLR has made
04:06
a very good job, in my opinion, at customizing it. So here is the page showing the collections available, and I can drill down into one, get the detail about one collection, then I can drill down and get a list of the items
04:25
in that collection, and eventually get into a single item with a map, with a footprint, the thumbnail, and all the assets for download of the data, or direct usage, because they might be linking to a cloud-optimized geotiff, and then I could stream directly from
04:44
that source. We also have a back-end REST API, so that you can go and manage the collections, the products, the granules, and so on, so that you can automate the management of the contents
05:03
of the database. Okay, say that I found what I want. I know which products I want. So now, how do I access that raster data? Well, as I said, stack and open search, they both have links, typically to services, but also to the unit elements that make up the product, so the assets.
05:26
GeoServer-wise, they might be located on the local file system, which is kind of the most common use case, but also the less interesting one, or they could be put on an HTTP server, on an S3 bucket, or Azure blob, or any cloud blob storage, and we can read directly from
05:47
those blob stores using the COG plugin, the cloud-optimized reader, which leverages the code structure. In GeoServer, we also have an older reader that could read literally any geotiff, no matter whether it was formatted as a COG or not. It's less efficient, but it's more general.
06:06
The COG plugin is smart enough to get the header of the file, and then just do HTTP range reads to just get the tiles that it needs without having to read anything else. It currently works against S3, HTTPS, Google storage. We are implementing it on Azure blob these
06:25
days, like it's going to be available in weeks. We might get local caching of the blob contents, which would be interesting for speed. If you're interested, please contact me. Of course, more blob storage options. No. Okay, I can get to the raster data.
06:47
How do I put it together? Typically, with an image mosaic. Typically, I've located a bunch of products, not just one, and I want to see them. We use the GeoServer Mosaikin machinery to
07:01
go and locate the data. The Mosaikin machinery is powerful enough that I can fetch and mosaic together data in different coordinate reference systems, different color models, different resolutions, which is, by the way, very important for, say, Sentinel-2, where each product is in a different UTM zone, and different bands have different resolutions,
07:21
and so on. This kind of freedom is actually very important to get things going. The Mosaikin index, typically, is stored in a database, like PostGIS, Oracle SQL Server, but it could also be located in a shapefile or other sources. I'm going to get there in a moment.
07:40
Well, it's typically looking more or less like a table, where I have the location of the image, its footprint, but also all the attributes that I would like to use for filtering and sorting and stacking the images. The time, the elevation, the runtime of the weather model, and so on. As I was saying, we have dimensions.
08:03
Dimensions are then recognized by OGC protocols. If I look at the WMS capabilities document or the elevation domain and so on, that the clients can then use to filter down the data that they want.
08:23
Interesting enough, we can use the stack slash open search database as an index for image Mosaik. If I ever implemented stack or open search in my system, there is a quick REST API that I can use to say, please make me a layer out of this collection. It literally
08:43
pipes directly into that database. What I'm working on, literally these days, I'm sort of halfway in the implementation, is the idea of having GeoServer connected to an external stack API, treat it as an index of sorts, and then being able to display on the map
09:04
the footprints, filter them, and then eventually flip to raster representation if the stack API is pointing to cogs, to cogs that I can access. A bunch of links, they are there for you to follow
09:20
when you get a hold of the presentation slides, if you are interested. With image Mosaik, we can do a bunch of interesting stuff. When you are Mosaik-ing together several products, you probably want to filter, and Mosaik-ing allows you to filter on any attributes in the index, but also decide the stacking. I want the color images on top,
09:42
or I want the most recent images on top, or less cloudy images on top. You can perform sorting, and if I'm talking to a stack API, hopefully it's implementing the sorting option, and I will literally flip the sort onto the stack API, get the items that I want, and
10:01
Mosaik-ing the imagery accordingly. Another interesting thing is coverage views, which is this idea that I might have several raster layers that represent maybe different bands or different components of a phenomenon, and stick them back together in a single raster that I can then use for visualization. This use case is showing taking two bands, which are U and
10:24
V, the projection of the wind vector on the two axes, and putting them back together so that I can do math on them and display wind barbs. Another use case is taking central image imagery and do false color imagery, or I'm going to get to it. I thought I had the NDVI slide,
10:47
but it's going to get there. Once I am ready to use the Mosaik, I'm ready to visualize the data. Visualize through WMS or to OGC API maps. WMS is going to export all the dimensions that
11:05
I have in the capabilities document. I will be able to quickly filter on time and elevation and custom dimensions through the URL. I will be able to do custom filtering using SQL filter or sorting and so on. All that I do on the WMS is going to be packed down into the original
11:26
index, which might be the stack API, and then I'm going to talk to the stack API and say, this filter, this sorting. We can do a bunch of rendering transformations, which is the idea of taking data and processing it quickly to get a different display. So contouring,
11:46
wind barbs, currents, but also NDVI on the fly. So we have our little map algebra language that we can stick into the styles to perform simple index calculations such as NDVI, NDWI,
12:04
and so on and so on. And a bunch of other links that you can use. Then if you are satisfied with the data and you actually want to download it, you can use WCS, which is the OGC protocol to do that. The WCS 2.0 implementation into
12:23
your server is complete. The protocol is same, simple to use. And well, in the description of the coverage, you get all the spatial and temporal information about the coverage. And we added a few vendor extensions in GeoServer so that we can specify the same
12:45
filtering and the same sorting that we applied in WMS so that I saw something and then I can download that particular mosaic with those products inside and get a geotiff out of it with the same structure. However, WCS has some limitations in that the requests are synchronous. So I'm
13:08
sending the request, waiting on the HP link, and maybe I'm waiting half an hour, and I'm not going to wait that long because something is going to time out my request. So what do I do about large downloads? I can use WPS. WPS is the OGC protocol that has
13:22
asynchronous request support built in. So we implemented in GeoServer a process that's called WPS download, which is designed to allow large downloads of both vector data and raster data. So I can hit a very large image mosaic and wait whatever time is needed and then sometime later
13:49
fetch 25 gigabytes worth of geotiff out of GeoServer without the timeouts. So of course, you're going to have this user interface telling the user, okay, I'm waiting for the
14:00
download to come back later. And at some point, it's going to notify that the download is available. We can also do large rendered maps. So maybe what you want to do is actually to to have a printable geotiff like 20,000 by 40,000 NDVI properly color the image,
14:21
which is also something that you wouldn't be able to do with get a map with WMS for the same reason. It's synchronous. So yes, I'm almost done. And so with WPS download, we can say, yeah, I want to download this very large map, and it's going to take whatever time I'm going to wait.
14:41
In addition to that, we can do long animations. So say that I have, like in this case, a Meteosat time series, and I wanted to make an MP4, a video out of it. With WPS download, I can also instruct the server to work on a certain layer and a certain amount of time range, build the
15:05
frames, build my animation, and after some time, I can download my MP4 with the animation. And with that said, I'm done.
Recommendations
Series of 351 media