We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

View Server - Cloud Native EO Data Access

00:00

Formal Metadata

Title
View Server - Cloud Native EO Data Access
Title of Series
Number of Parts
156
Author
Contributors
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
The View Server (VS) software project was developed for cloud native geospatial data access. This includes functionality to browse, search, transform and download Earth Observation (EO) and other geospatial data via a range of OGC compliant and other standardized interfaces such as STAC, OpenSearch, WMS, WMTS and WCS and their OGC API successors. Based on EOxServer, powerful rendering capabilities are built in, allowing on-the-fly data transformation and colorization for exploring datasets, which can subsequently be tiled and cached via the built in MapCache. Data is ingested via feature rich components to harvest, enrich metadata, preprocess and pre-seed caches, to offer a performance optimized and flexible rendering of the data via its service endpoints. Using the harvested and enriched metadata, CQL filters can be employed to filter down the records to be visualized or searched, whereas an expression language is used to flexibly define the renderings of either RGB or color-scaled outputs. As a cloud native component, View Server allows various storage systems, such as OpenStack Swift and S3 and can be installed as a system in a kubernetes (via Helm Charts) or Docker Swarm environments. In the recently concluded Earth Observation Exploitation Platform Common Architecture (EOEPCA), View Server was both employed in the global and user workspace data access contexts. In ongoing developments, View Server will be made compatible with the other components of the eoAPI, allowing it to share a common data model based on STAC and as an interchangeable component in an eoAPI deployment but with all rendering features remaining. View Server is also used in other operational deployments for data preprocessing, access and visualisation. Those include the ESA Payload Data Ground Segment Software Applications (PDGS) or Copernicus Space Component Data Access system (CSCDA) for a vast number of active and discontinued optical and SAR satellite missions. Lastly, it supports data access for our Agricultural Area Monitoring Systems (AMS).
Keywords
127
Streaming mediaOffice suiteService (economics)View (database)Server (computing)Point cloudView (database)Server (computing)Projective planeSoftwareLecture/Conference
Point cloudServer (computing)View (database)Service (economics)Demo (music)ExistenceSatelliteServer (computing)View (database)MereologyPhysical systemDemo (music)SoftwarePoint (geometry)QR codeSource codeOpen sourceComputer animationLecture/ConferenceMeeting/Interview
SatellitePoint cloudData storage deviceFile formatOpen sourceVolumenvisualisierungService (economics)Software maintenanceConfiguration spaceImplementationView (database)Client (computing)Server (computing)Medical imagingSoftwarePoint cloudProjective planeSatelliteGoodness of fitCASE <Informatik>ImplementationMathematical optimizationSoftware maintenanceData managementConfiguration spaceStandard deviationFile formatInterface (computing)MereologyComponent-based software engineeringField (computer science)Internet service providerComputer animation
Web pageSatelliteArtistic renderingParameter (computer programming)View (database)Client (computing)Server (computing)Computer animation
Client (computing)VolumenvisualisierungParameter (computer programming)File formatCache (computing)Type theoryProduct (business)StatisticsComputer-generated imageryStandard deviationAxonometric projectionMusical ensembleRegulärer Ausdruck <Textverarbeitung>MathematicsComputer configurationRange (statistics)Variable (mathematics)Point cloudVector spacePoint cloudMedical imagingSatelliteMusical ensembleArtistic renderingGraph coloringSelectivity (electronic)Server (computing)Auditory maskingProjective planeVector spaceProduct (business)MathematicsInteractive televisionView (database)ResultantPhysical systemLevel (video gaming)SubsetCartesian coordinate systemVolumenvisualisierungParameter (computer programming)Core dumpExpressionClient (computing)Statistical dispersionSoftwareOrder (biology)Data structureOpen sourceTesselationCache (computing)State observerOperator (mathematics)Set (mathematics)MultilaterationBitSource codeObservational studyComputer animationProgram flowchart
Web pageFunctional (mathematics)Point cloudChromatisches PolynomOperator (mathematics)Graph coloringMusical ensemblePunched cardImage resolution
Service (economics)Web pageScalabilityFluid staticsProcess (computing)Virtual realityGamma functionScale (map)Musical ensembleRegulärer Ausdruck <Textverarbeitung>Extension (kinesiology)VolumenvisualisierungMetadataImage warpingDimensional analysisUtility softwareDiscrete element methodMathematicsFunction (mathematics)Codierung <Programmierung>DemosceneCache (computing)DatabaseClient (computing)View (database)Mountain passServer (computing)Distribution (mathematics)Digital filterWeb browserUniform resource locatorLink (knot theory)Product (business)Medical imagingIterationVolumenvisualisierungSelectivity (electronic)Cartesian coordinate systemScaling (geometry)Order (biology)Atomic numberDemosceneComputer fileExtension (kinesiology)Source codeStack (abstract data type)ResultantPoint (geometry)Right angleForm (programming)BefehlsprozessorVirtualizationOperator (mathematics)File formatCore dumpCuboidServer (computing)View (database)MathematicsMusical ensembleSingle-precision floating-point formatPoint cloudArtistic renderingProjective planeGraph (mathematics)Arithmetic progressionSimilarity (geometry)Structural loadElectric generator1 (number)Term (mathematics)Internet service providerClient (computing)SoftwareTesselationElectronic mailing listMultiplication signProduct (business)Distribution (mathematics)Cache (computing)Subject indexingLibrary (computing)Graph coloringLevel (video gaming)Functional (mathematics)MereologyDirection (geometry)Range (statistics)NP-hardProcess (computing)AdditionCubeSoftware developerHistogramZoom lensDebuggerGreatest elementInterface (computing)BitDefault (computer science)Front and back endsDynamical systemWeb pageGoodness of fitDatabasePrice indexImage warpingInformationGraphical user interfaceMetadataComputer animationProgram flowchart
Web pageDiscrete element methodNominal numberInheritance (object-oriented programming)Uniform resource nameLink (knot theory)MetadataMereologyLocal ringData storage devicePoint cloudObject (grammar)FluxLoginRepository (publishing)Axonometric projectionClient (computing)Service (economics)Modul <Datentyp>Image registrationProduct (business)Directed setCache (computing)Demo (music)Instance (computer science)Performance appraisalComputer-generated imagerySingle-precision floating-point formatInformationOrder (biology)Product (business)File archiverData storage devicePoint cloudDimensional analysisMultiplication signWeb browserData transmissionMedical imagingServer (computing)Level (video gaming)Zoom lensClient (computing)ChainGeometryCache (computing)Source codeDatabaseMereologyMechanism designDirektes ProduktCartesian coordinate systemImage registrationCommunications protocolView (database)Physical systemStack (abstract data type)Projective planeOpen setSatelliteSession Initiation ProtocolSubject indexingMeasurementField (computer science)Computer animation
Web pageComputer-generated imageryDemo (music)Performance appraisalInstance (computer science)Server (computing)View (database)Electronic program guideOperator (mathematics)Software developerConfiguration spaceData storage deviceFile formatSource codeProduct (business)Type theoryMusical ensembleVisualization (computer graphics)SubsetCache (computing)Texture mappingPhysical systemMereologyProxy serverCore dumpDefault (computer science)Domain nameInterface (computing)Subject indexingField (computer science)MeasurementCartesian coordinate systemSet (mathematics)Gene clusterServer (computing)View (database)Source codeDefault (computer science)Computer animation
Visualization (computer graphics)Musical ensembleCache (computing)SubsetTexture mappingType theoryProduct (business)Web pageSoftware developerServer (computing)Electronic program guideOperator (mathematics)Configuration spaceMereologyProxy serverPhysical systemDefault (computer science)Core dumpData storage deviceFile formatSource codeView (database)Network topologyIntegrated development environmentCodePersonal digital assistantService (economics)Functional (mathematics)File formatPoint cloudServer (computing)Validity (statistics)SatelliteMappingVector spaceLevel (video gaming)Raster graphicsOrthogonalityMathematical optimizationSet (mathematics)Configuration spaceResultantLink (knot theory)Data structureData conversionIntegrated development environment1 (number)Order (biology)CodeMultiplication signPreprocessorInterface (computing)Repository (publishing)Maxima and minimaStack (abstract data type)Point (geometry)Presentation of a groupSoftware developerBitCartesian coordinate systemControl flowProcess (computing)Open sourceMusical ensembleMereologyGradient2 (number)AreaView (database)Computer animationLecture/ConferenceMeeting/Interview
ImplementationOrder (biology)Fluid staticsEndliche ModelltheorieInterface (computing)Mixed realityRight angleLibrary catalogSystem programmingLecture/Conference
Right angleLecture/Conference
Computer-assisted translationInclusion mapComputer animation
Transcript: English(auto-generated)
Thank you very much. Hello everyone as well from me and Fabian. We will be co-hosting this talk. My name is Lubomir Dolezal and as together with EoX, we are mostly working on projects from European Space Agency and
we are going to present a talk about a piece of software that we use quite for quite many of them called View Server and the main point of it is that it provides us with WMS and WCS services for satellite data. That's what the main purpose of it is.
On the QR code you can have a look at our demo deployment. You can browse through it throughout the talk as I will be and we will be discussing various parts of the whole system. First I will start with a question. Who of you have ever
heard about a software called View Server? Hands up, a few. Who of you have used View Server? None, because otherwise you would not really be here, right? Because you would already know everything. In the talk we will describe what it is composed of,
what it can do for you, how to deploy it and because it's an open source piece of software, how you can actually use it and contribute to it. That would be the ultimate goal if one of you or more actually become contributors to the View Server software stack. First, why would you use it? View Server is a Python based piece of software components, which is
used for managing and providing OGC compliant services as WMS, WCS for your satellite data that you want to serve to people and to clients via standard interfaces that everybody understands.
We have tried to provide access to our services mainly with respect to cloud optimized formats because that's how we get the best browsing performance. We have of course tried to go with the microservice architecture
because it helps us to deploy only parts of the services that we actually need for a concrete client. The whole stack is open source. It's an MIT license. You are free to use it however you see fit. How we work on it is in two ways.
First, we provide the services which the View Server fields, which means the customer is not really interested what the software behind is as long as there is a WMS endpoint and that serves images of satellite data. All good. In other cases,
we are providing the configuration and the maintenance and implementation of new features themselves. So both of the cases are present among our projects for European Space Agency. This is the ultimate goal when you deploy a server, when you deploy View Server, this is what you will probably end up. It will be
WMTS or WMS rendering of your satellite data in various predefined styles and the client will be able to interact with it using WMS parameters if they see a need to update the rendering themselves. So how do we actually get to it?
Well, there are a few components of this system. There is usually a client that we also ship together with View Server. There is some cache component in order to precede tiles and then serve them. We are talking about raster tiles. There is the rendering engine itself,
which is the core of the whole View Server and we need to get the data into the system somehow. So first, the rendering. Currently, still, the rendering is based on an open source piece of software called EoX Server, which is a Python based Django application which provides the endpoints and the service handling. The rendering itself is
using Map Server underneath another open source piece of software that you've probably already heard about. As for the capabilities themselves, the core of the EoX Server that we use mostly is the WMS endpoint with the flexible rendering. Basically, what happens is you define how your satellite data structure
matches to the layers that you want to serve in the end. That means which band composition and which value stretches will in the end contribute to the RGB result that gets rendered. This means
usually we want some kind of flexibility. We have three custom WMS parameters for band selection, setting the stretch and using some kind of custom rendering variables for higher interactivity with the clients. The rendering stack is quite configurable in a way that it allows a lot of mathematic expressions using the
rendering using the expressions inside the request itself. We allow masking or CQL for selection of only a subset of the images and you will see examples of that a little bit later.
Exactly here. On the left side you see a false color composite, which is first you define a set of bands that will be used for the red, green, blue result of the final image. With a custom stretch, it's the usual
operation that all of us want to do when serving Earth Observation data. The middle one is showing CQL filtering based on cloud coverage. So when we register together with our metadata, the cloud coverage, we can use it in the WMS request for filtering just a subset of products. The last one is an example from an existing project using vector masks also for
masking out clouds. Another example is using the punch sharpening operation on the fly, using the GDAL punch sharpen functionality underneath for giving really from panchromatic band and the true color, a low resolution band, the punch sharpened image.
Thanks, Luba. Yeah, Luba talked about the previous iterations of our renderer. We also are working currently on the next generation, which is called Stackture and Teraviz.
Stackture, Stack stands in the name. This is why we chose it. Quick hint, use project names which are easy for not being handled by autocorrect. Stackture is not such of those those names. Yeah, why did we choose to
the next generation of APIs out there? So this is now totally stack-based, so we don't even ship our own database anymore, but we simply connect to a stack API that we then use as a provider for scenes to then render.
It is also modernized in terms of interfaces that it provides. It's also serving now the OGC APIs, not just the previous OWS ones. It is still somewhat early development. It's still in the making, but we are making fast progress and our aim is to
yeah, at some point replace the OGC server as the main rendering engine. As the name implies, it consists of two main components, Stackture, which is basically the front-end facing part of it, and Teraviz, which is doing the actual rendering work so that the hard work. The idea is that we have a stateless thing, so it simply connects to the stack API or other stack sources.
It's also very much scalable, so we can also individually scale the front-end or the back-end part, Teraviz or Stackture. Because Teraviz is doing the hard work, we can scale this individually, so this will help in cloud developments when we are serving thousands of clients. Also, it is capable of doing a serving of DataCube, so multi-dimensional data by default.
It's also accessible like the other part of the ViewServer project and also MIT licensed. Okay, how does this look like in a deployment form? So we can see here on the right-hand side, we see all the interfaces, OGC, OWS interfaces, also open search, and we also want to provide a stack API at some point.
On the left-hand side, we can see the whole sources that we connect to, so PGStack for example, or Stack API, so we try to provide as much as, so a diverse range of
stack sources. Stackture is basically the entry point. It will translate the request and the stack items to a custom workflow format, which is JSON-based, which is sent to Teraviz, which interprets it, and then send out the result. We are making use of various stack extensions and also the core format obviously, but stack extensions, so for example, using their
the virtual assets extension in order to provide, for example, band composition or band math on the fly. We're using the render extension in order to make like color scale application and so on, or band selection, or like
view saturation, yeah, corrections and so on. And the classification extension we're using for color mesh, for example, you can see here an SEL file, so this is a sidecar of a Sentinel-2 scene, and there's this common color scale that we simply apply, so we can also render this. I briefly mentioned what is a Teraviz workflow. This is
JSON-based format, and this basically describes the atomic operations in order to produce the result. Each node in this direct acyclic graph, or short DAC, represents an atomic operation, like load the data, warp the data, or something similar, like arithmetic operation and so on.
In order to not bloat the the DAC too much, we also allow reusable parts of this, which you can see on the right-hand side, so this yellow box is basically reused multiple times, like you would call a function. Also,
since each node is basically an atomic operation, it's very easy to parallelize, so we yeah, we can scale this quite well with multiple CPUs, we try to broaden the work of each individual step, and we can do many steps at once, and we are smart enough to then collect and combine the results back together to a single product.
Right, the next component is the cache. The cache is based on the very popular map cache library, map cache software, which does all the heavy-duty work. We are making use of the tile, the time index DB, so where we basically know, okay, which time indices are available,
we query that for better caching, so that we know at a request this is targeting these and these and these tiles, and we can simply combine them back to a single result. So we are pre-configuring it, we are basically don't change anything with map cache, it's a good piece of software. On the other hand, we have a client which is connecting to either the cache or the dynamic renderer.
This is based on a software called UXE, it's a little bit rusty on the edges, but it's still working perfectly, so you can get a nice rendered map interface, where you can simply query the data, look for collections. We also in the bottom have
something, I think we call it this time slider, it's basically a histogram over time, and you can see just the temporal distribution of the data, so it's also quite nice that you can zoom into this time slider like a map and zoom out and pan around. You can also like discover the products that you have searched for, you can select them, you can do
additional downloading, processing and so on, all through this GUI. This is a details page, so we can simply click on a single product in order to get some more details, metadata information and nicely rendered views. The last part is how to actually get data in.
This is the ingestion chain, this part, it's consisting of multiple parts that you can basically pick and choose. We have two different mechanisms, we have the push-based mechanism which is called the ingestors, you simply push items into it and it will then try to register it. But the other way is the harvester, so we can harvest upstream sources like open search or stack APIs and this will pull them
one by one into the system and register them. If necessary, we can also have a pre-processing chain where we can make for example cloud-optimized geotiff out of it in order to improve the rendering performance, then it goes through the registrar and
basically registered it to the database. I think we also have some preceding facilities in order to have certain zoom levels already in the cache for the clients to use. Right, I think that was the ingestion. Yeah, I'll hand over to Lube again. Okay, and I will finish with listing the capabilities in a unified way. A lot of these things were already mentioned.
We are quite successful in scaling this for large satellite archives, which means hundreds of thousands or millions of products usually split into multiple collections, so it's not one-to-one, but the time dimension helps us to really granularly pick whatever the client should
access and we render that. We are heavily using cloud storage, so we have accessors for both S3 and OpenStack Swift. We are actively using both.
The local access is kind of a relic from history. Many rendering scenarios are supported depending on what the project actually needs. We can configure it. The direct product registration is used when already the archive contains cloud-optimized geotaves because we use stack as an internal data transfer protocol.
Therefore, if there is already a stack, we would be able to register it directly. The web client is kind of an add-on that is not always used because usually just the WMS and UNTS services are used. One deployment example is an agricultural application
using the WMS directly for individual image chips for individual dates, so imagine you have a parcel somewhere on your field and you want to measure NDVI index over it throughout the whole half a year or a year. This is an interface that we have ready for that for our agricultural domain
application. And coming back to the community, you can use ViewServer. How would you do that? We have a Helm chart containing a reasonable set of default values for deploying on smaller or even larger clusters, which means we are using Kubernetes as our main source of deployment,
but we also have some conversions to Docker Swarm for those environments who, for example, don't have managed Kubernetes. Then it just comes to defining how your satellite data structure maps to the rendering result that you want to provide and how much
pre-processing you need or want to do. There are several links to our documentation and for the Helm chart itself, we have a set of schemas so that whenever you are creating your deployment and you have an invalid structure of your values, it will already scream at you early and you don't need to deploy your failing environment over and over again.
How you can contribute? First, let us know what you would want to use ViewServer for. Maybe the capabilities are already supported and it's just a matter of configuration. You can also try the minimum example of the UX server itself without all the nice things of your server itself.
And the whole code is open source. It's a Python-based application, very accessible, I would say to developers. And if you have any missing features, like for example, I don't know what would be missing, but if you have missing features, also let us know in the issue tracker
the repository is public, even though it's on our company, GitLab. I'm ending a little bit early. Thank you very much for your attention. Wow! You're giving some time back. That's so nice. So we have some time for questions.
So I think we have plenty of time, so please. Thank you for the presentation. You mentioned that you are publishing with some OGC APIs and I was curious to know which ones you are using. So right now we support OGC API maps and coverages in the making. Yeah.
I think we have ties at some point as well, but I think ties is like orthogonal to maps and coverages. This is great work.
Any thoughts around supporting vector data and rendering vector data? So we're using vector data like in support of our raster data, so we're mainly focused on raster data. We use it, for example, for masking out images like for the
valid areas, for example, or the footprint. We can show the footprint as well. But yeah, it's a second-grade citizen to us. We are, yeah, raster data.
You managed S3 support. Do you need to somehow pre-process the data to make it viable to pull it from S3, or how does it work right now? Well, we are using the great capabilities of GDAL. Thanks for that, Ivan. So it works best with Cloud Optimized formats, obviously, but it will also work with any other formats because this is how GDAL works.
Yeah, this is why we have to pre-process in order to like make pre-selections, use Cloud Optimized formats like GTFs. Yeah, it works best that way, but it's not a requirement.
So I guess for the actual rendering part, you're moving away from map server, right? Yes, unfortunately, yeah. It is what it is. What is the actual way that you're rendering then? Are you using like GDAL?
Yeah, we're using the Python API of GDAL. We were thinking of like using Rasterio, but like GDAL and Python API still has the best set of features, like all of the pan sharpening and the dam functionality that we also provide. It's just available through the Python API, so we're using that. Thank you.
Hello, have you considered using
OTC RP processes between Stecture and Theraviz? Yes, yes, we did. So yeah, we decided to define our own workflow formats, but we're also thinking of like standardizing the interface between Stecture and Theraviz, like this would be a great idea to use processes.
So you mentioned you are using both the Stack API and Stack Static Catalog, right? Are you using any tools or is it native implementation that you have there?
That's actually a good question, but I think we are using our own interfaces in order to access them. PyStack might be in the mix somewhere. I think we're using PyStack items as models in between the systems, but I think we're doing our own requests. I'm not too sure about that, to be honest. Thank you.