View Server - Cloud Native EO Data Access
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 156 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/68568 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
FOSS4G Europe 2024 Tartu147 / 156
6
33
35
53
55
59
61
67
70
87
97
99
102
103
104
105
107
111
121
122
123
124
125
126
127
128
134
144
150
151
155
00:00
Streaming mediaOffice suiteService (economics)View (database)Server (computing)Point cloudView (database)Server (computing)Projective planeSoftwareLecture/Conference
00:26
Point cloudServer (computing)View (database)Service (economics)Demo (music)ExistenceSatelliteServer (computing)View (database)MereologyPhysical systemDemo (music)SoftwarePoint (geometry)QR codeSource codeOpen sourceComputer animationLecture/ConferenceMeeting/Interview
01:37
SatellitePoint cloudData storage deviceFile formatOpen sourceVolumenvisualisierungService (economics)Software maintenanceConfiguration spaceImplementationView (database)Client (computing)Server (computing)Medical imagingSoftwarePoint cloudProjective planeSatelliteGoodness of fitCASE <Informatik>ImplementationMathematical optimizationSoftware maintenanceData managementConfiguration spaceStandard deviationFile formatInterface (computing)MereologyComponent-based software engineeringField (computer science)Internet service providerComputer animation
03:12
Web pageSatelliteArtistic renderingParameter (computer programming)View (database)Client (computing)Server (computing)Computer animation
03:42
Client (computing)VolumenvisualisierungParameter (computer programming)File formatCache (computing)Type theoryProduct (business)StatisticsComputer-generated imageryStandard deviationAxonometric projectionMusical ensembleRegulärer Ausdruck <Textverarbeitung>MathematicsComputer configurationRange (statistics)Variable (mathematics)Point cloudVector spacePoint cloudMedical imagingSatelliteMusical ensembleArtistic renderingGraph coloringSelectivity (electronic)Server (computing)Auditory maskingProjective planeVector spaceProduct (business)MathematicsInteractive televisionView (database)ResultantPhysical systemLevel (video gaming)SubsetCartesian coordinate systemVolumenvisualisierungParameter (computer programming)Core dumpExpressionClient (computing)Statistical dispersionSoftwareOrder (biology)Data structureOpen sourceTesselationCache (computing)State observerOperator (mathematics)Set (mathematics)MultilaterationBitSource codeObservational studyComputer animationProgram flowchart
06:25
Web pageFunctional (mathematics)Point cloudChromatisches PolynomOperator (mathematics)Graph coloringMusical ensemblePunched cardImage resolution
06:45
Service (economics)Web pageScalabilityFluid staticsProcess (computing)Virtual realityGamma functionScale (map)Musical ensembleRegulärer Ausdruck <Textverarbeitung>Extension (kinesiology)VolumenvisualisierungMetadataImage warpingDimensional analysisUtility softwareDiscrete element methodMathematicsFunction (mathematics)Codierung <Programmierung>DemosceneCache (computing)DatabaseClient (computing)View (database)Mountain passServer (computing)Distribution (mathematics)Digital filterWeb browserUniform resource locatorLink (knot theory)Product (business)Medical imagingIterationVolumenvisualisierungSelectivity (electronic)Cartesian coordinate systemScaling (geometry)Order (biology)Atomic numberDemosceneComputer fileExtension (kinesiology)Source codeStack (abstract data type)ResultantPoint (geometry)Right angleForm (programming)BefehlsprozessorVirtualizationOperator (mathematics)File formatCore dumpCuboidServer (computing)View (database)MathematicsMusical ensembleSingle-precision floating-point formatPoint cloudArtistic renderingProjective planeGraph (mathematics)Arithmetic progressionSimilarity (geometry)Structural loadElectric generator1 (number)Term (mathematics)Internet service providerClient (computing)SoftwareTesselationElectronic mailing listMultiplication signProduct (business)Distribution (mathematics)Cache (computing)Subject indexingLibrary (computing)Graph coloringLevel (video gaming)Functional (mathematics)MereologyDirection (geometry)Range (statistics)NP-hardProcess (computing)AdditionCubeSoftware developerHistogramZoom lensDebuggerGreatest elementInterface (computing)BitDefault (computer science)Front and back endsDynamical systemWeb pageGoodness of fitDatabasePrice indexImage warpingInformationGraphical user interfaceMetadataComputer animationProgram flowchart
12:50
Web pageDiscrete element methodNominal numberInheritance (object-oriented programming)Uniform resource nameLink (knot theory)MetadataMereologyLocal ringData storage devicePoint cloudObject (grammar)FluxLoginRepository (publishing)Axonometric projectionClient (computing)Service (economics)Modul <Datentyp>Image registrationProduct (business)Directed setCache (computing)Demo (music)Instance (computer science)Performance appraisalComputer-generated imagerySingle-precision floating-point formatInformationOrder (biology)Product (business)File archiverData storage devicePoint cloudDimensional analysisMultiplication signWeb browserData transmissionMedical imagingServer (computing)Level (video gaming)Zoom lensClient (computing)ChainGeometryCache (computing)Source codeDatabaseMereologyMechanism designDirektes ProduktCartesian coordinate systemImage registrationCommunications protocolView (database)Physical systemStack (abstract data type)Projective planeOpen setSatelliteSession Initiation ProtocolSubject indexingMeasurementField (computer science)Computer animation
15:29
Web pageComputer-generated imageryDemo (music)Performance appraisalInstance (computer science)Server (computing)View (database)Electronic program guideOperator (mathematics)Software developerConfiguration spaceData storage deviceFile formatSource codeProduct (business)Type theoryMusical ensembleVisualization (computer graphics)SubsetCache (computing)Texture mappingPhysical systemMereologyProxy serverCore dumpDefault (computer science)Domain nameInterface (computing)Subject indexingField (computer science)MeasurementCartesian coordinate systemSet (mathematics)Gene clusterServer (computing)View (database)Source codeDefault (computer science)Computer animation
16:06
Visualization (computer graphics)Musical ensembleCache (computing)SubsetTexture mappingType theoryProduct (business)Web pageSoftware developerServer (computing)Electronic program guideOperator (mathematics)Configuration spaceMereologyProxy serverPhysical systemDefault (computer science)Core dumpData storage deviceFile formatSource codeView (database)Network topologyIntegrated development environmentCodePersonal digital assistantService (economics)Functional (mathematics)File formatPoint cloudServer (computing)Validity (statistics)SatelliteMappingVector spaceLevel (video gaming)Raster graphicsOrthogonalityMathematical optimizationSet (mathematics)Configuration spaceResultantLink (knot theory)Data structureData conversionIntegrated development environment1 (number)Order (biology)CodeMultiplication signPreprocessorInterface (computing)Repository (publishing)Maxima and minimaStack (abstract data type)Point (geometry)Presentation of a groupSoftware developerBitCartesian coordinate systemControl flowProcess (computing)Open sourceMusical ensembleMereologyGradient2 (number)AreaView (database)Computer animationLecture/ConferenceMeeting/Interview
21:10
ImplementationOrder (biology)Fluid staticsEndliche ModelltheorieInterface (computing)Mixed realityRight angleLibrary catalogSystem programmingLecture/Conference
21:37
Right angleLecture/Conference
21:54
Computer-assisted translationInclusion mapComputer animation
Transcript: English(auto-generated)
00:01
Thank you very much. Hello everyone as well from me and Fabian. We will be co-hosting this talk. My name is Lubomir Dolezal and as together with EoX, we are mostly working on projects from European Space Agency and
00:24
we are going to present a talk about a piece of software that we use quite for quite many of them called View Server and the main point of it is that it provides us with WMS and WCS services for satellite data. That's what the main purpose of it is.
00:45
On the QR code you can have a look at our demo deployment. You can browse through it throughout the talk as I will be and we will be discussing various parts of the whole system. First I will start with a question. Who of you have ever
01:00
heard about a software called View Server? Hands up, a few. Who of you have used View Server? None, because otherwise you would not really be here, right? Because you would already know everything. In the talk we will describe what it is composed of,
01:21
what it can do for you, how to deploy it and because it's an open source piece of software, how you can actually use it and contribute to it. That would be the ultimate goal if one of you or more actually become contributors to the View Server software stack. First, why would you use it? View Server is a Python based piece of software components, which is
01:45
used for managing and providing OGC compliant services as WMS, WCS for your satellite data that you want to serve to people and to clients via standard interfaces that everybody understands.
02:02
We have tried to provide access to our services mainly with respect to cloud optimized formats because that's how we get the best browsing performance. We have of course tried to go with the microservice architecture
02:22
because it helps us to deploy only parts of the services that we actually need for a concrete client. The whole stack is open source. It's an MIT license. You are free to use it however you see fit. How we work on it is in two ways.
02:40
First, we provide the services which the View Server fields, which means the customer is not really interested what the software behind is as long as there is a WMS endpoint and that serves images of satellite data. All good. In other cases,
03:01
we are providing the configuration and the maintenance and implementation of new features themselves. So both of the cases are present among our projects for European Space Agency. This is the ultimate goal when you deploy a server, when you deploy View Server, this is what you will probably end up. It will be
03:23
WMTS or WMS rendering of your satellite data in various predefined styles and the client will be able to interact with it using WMS parameters if they see a need to update the rendering themselves. So how do we actually get to it?
03:43
Well, there are a few components of this system. There is usually a client that we also ship together with View Server. There is some cache component in order to precede tiles and then serve them. We are talking about raster tiles. There is the rendering engine itself,
04:01
which is the core of the whole View Server and we need to get the data into the system somehow. So first, the rendering. Currently, still, the rendering is based on an open source piece of software called EoX Server, which is a Python based Django application which provides the endpoints and the service handling. The rendering itself is
04:24
using Map Server underneath another open source piece of software that you've probably already heard about. As for the capabilities themselves, the core of the EoX Server that we use mostly is the WMS endpoint with the flexible rendering. Basically, what happens is you define how your satellite data structure
04:45
matches to the layers that you want to serve in the end. That means which band composition and which value stretches will in the end contribute to the RGB result that gets rendered. This means
05:00
usually we want some kind of flexibility. We have three custom WMS parameters for band selection, setting the stretch and using some kind of custom rendering variables for higher interactivity with the clients. The rendering stack is quite configurable in a way that it allows a lot of mathematic expressions using the
05:22
rendering using the expressions inside the request itself. We allow masking or CQL for selection of only a subset of the images and you will see examples of that a little bit later.
05:42
Exactly here. On the left side you see a false color composite, which is first you define a set of bands that will be used for the red, green, blue result of the final image. With a custom stretch, it's the usual
06:00
operation that all of us want to do when serving Earth Observation data. The middle one is showing CQL filtering based on cloud coverage. So when we register together with our metadata, the cloud coverage, we can use it in the WMS request for filtering just a subset of products. The last one is an example from an existing project using vector masks also for
06:24
masking out clouds. Another example is using the punch sharpening operation on the fly, using the GDAL punch sharpen functionality underneath for giving really from panchromatic band and the true color, a low resolution band, the punch sharpened image.
06:48
Thanks, Luba. Yeah, Luba talked about the previous iterations of our renderer. We also are working currently on the next generation, which is called Stackture and Teraviz.
07:02
Stackture, Stack stands in the name. This is why we chose it. Quick hint, use project names which are easy for not being handled by autocorrect. Stackture is not such of those those names. Yeah, why did we choose to
07:25
the next generation of APIs out there? So this is now totally stack-based, so we don't even ship our own database anymore, but we simply connect to a stack API that we then use as a provider for scenes to then render.
07:42
It is also modernized in terms of interfaces that it provides. It's also serving now the OGC APIs, not just the previous OWS ones. It is still somewhat early development. It's still in the making, but we are making fast progress and our aim is to
08:00
yeah, at some point replace the OGC server as the main rendering engine. As the name implies, it consists of two main components, Stackture, which is basically the front-end facing part of it, and Teraviz, which is doing the actual rendering work so that the hard work. The idea is that we have a stateless thing, so it simply connects to the stack API or other stack sources.
08:22
It's also very much scalable, so we can also individually scale the front-end or the back-end part, Teraviz or Stackture. Because Teraviz is doing the hard work, we can scale this individually, so this will help in cloud developments when we are serving thousands of clients. Also, it is capable of doing a serving of DataCube, so multi-dimensional data by default.
08:46
It's also accessible like the other part of the ViewServer project and also MIT licensed. Okay, how does this look like in a deployment form? So we can see here on the right-hand side, we see all the interfaces, OGC, OWS interfaces, also open search, and we also want to provide a stack API at some point.
09:06
On the left-hand side, we can see the whole sources that we connect to, so PGStack for example, or Stack API, so we try to provide as much as, so a diverse range of
09:20
stack sources. Stackture is basically the entry point. It will translate the request and the stack items to a custom workflow format, which is JSON-based, which is sent to Teraviz, which interprets it, and then send out the result. We are making use of various stack extensions and also the core format obviously, but stack extensions, so for example, using their
09:47
the virtual assets extension in order to provide, for example, band composition or band math on the fly. We're using the render extension in order to make like color scale application and so on, or band selection, or like
10:01
view saturation, yeah, corrections and so on. And the classification extension we're using for color mesh, for example, you can see here an SEL file, so this is a sidecar of a Sentinel-2 scene, and there's this common color scale that we simply apply, so we can also render this. I briefly mentioned what is a Teraviz workflow. This is
10:23
JSON-based format, and this basically describes the atomic operations in order to produce the result. Each node in this direct acyclic graph, or short DAC, represents an atomic operation, like load the data, warp the data, or something similar, like arithmetic operation and so on.
10:44
In order to not bloat the the DAC too much, we also allow reusable parts of this, which you can see on the right-hand side, so this yellow box is basically reused multiple times, like you would call a function. Also,
11:00
since each node is basically an atomic operation, it's very easy to parallelize, so we yeah, we can scale this quite well with multiple CPUs, we try to broaden the work of each individual step, and we can do many steps at once, and we are smart enough to then collect and combine the results back together to a single product.
11:22
Right, the next component is the cache. The cache is based on the very popular map cache library, map cache software, which does all the heavy-duty work. We are making use of the tile, the time index DB, so where we basically know, okay, which time indices are available,
11:41
we query that for better caching, so that we know at a request this is targeting these and these and these tiles, and we can simply combine them back to a single result. So we are pre-configuring it, we are basically don't change anything with map cache, it's a good piece of software. On the other hand, we have a client which is connecting to either the cache or the dynamic renderer.
12:05
This is based on a software called UXE, it's a little bit rusty on the edges, but it's still working perfectly, so you can get a nice rendered map interface, where you can simply query the data, look for collections. We also in the bottom have
12:24
something, I think we call it this time slider, it's basically a histogram over time, and you can see just the temporal distribution of the data, so it's also quite nice that you can zoom into this time slider like a map and zoom out and pan around. You can also like discover the products that you have searched for, you can select them, you can do
12:43
additional downloading, processing and so on, all through this GUI. This is a details page, so we can simply click on a single product in order to get some more details, metadata information and nicely rendered views. The last part is how to actually get data in.
13:02
This is the ingestion chain, this part, it's consisting of multiple parts that you can basically pick and choose. We have two different mechanisms, we have the push-based mechanism which is called the ingestors, you simply push items into it and it will then try to register it. But the other way is the harvester, so we can harvest upstream sources like open search or stack APIs and this will pull them
13:26
one by one into the system and register them. If necessary, we can also have a pre-processing chain where we can make for example cloud-optimized geotiff out of it in order to improve the rendering performance, then it goes through the registrar and
13:40
basically registered it to the database. I think we also have some preceding facilities in order to have certain zoom levels already in the cache for the clients to use. Right, I think that was the ingestion. Yeah, I'll hand over to Lube again. Okay, and I will finish with listing the capabilities in a unified way. A lot of these things were already mentioned.
14:08
We are quite successful in scaling this for large satellite archives, which means hundreds of thousands or millions of products usually split into multiple collections, so it's not one-to-one, but the time dimension helps us to really granularly pick whatever the client should
14:27
access and we render that. We are heavily using cloud storage, so we have accessors for both S3 and OpenStack Swift. We are actively using both.
14:41
The local access is kind of a relic from history. Many rendering scenarios are supported depending on what the project actually needs. We can configure it. The direct product registration is used when already the archive contains cloud-optimized geotaves because we use stack as an internal data transfer protocol.
15:04
Therefore, if there is already a stack, we would be able to register it directly. The web client is kind of an add-on that is not always used because usually just the WMS and UNTS services are used. One deployment example is an agricultural application
15:22
using the WMS directly for individual image chips for individual dates, so imagine you have a parcel somewhere on your field and you want to measure NDVI index over it throughout the whole half a year or a year. This is an interface that we have ready for that for our agricultural domain
15:44
application. And coming back to the community, you can use ViewServer. How would you do that? We have a Helm chart containing a reasonable set of default values for deploying on smaller or even larger clusters, which means we are using Kubernetes as our main source of deployment,
16:07
but we also have some conversions to Docker Swarm for those environments who, for example, don't have managed Kubernetes. Then it just comes to defining how your satellite data structure maps to the rendering result that you want to provide and how much
16:21
pre-processing you need or want to do. There are several links to our documentation and for the Helm chart itself, we have a set of schemas so that whenever you are creating your deployment and you have an invalid structure of your values, it will already scream at you early and you don't need to deploy your failing environment over and over again.
16:43
How you can contribute? First, let us know what you would want to use ViewServer for. Maybe the capabilities are already supported and it's just a matter of configuration. You can also try the minimum example of the UX server itself without all the nice things of your server itself.
17:05
And the whole code is open source. It's a Python-based application, very accessible, I would say to developers. And if you have any missing features, like for example, I don't know what would be missing, but if you have missing features, also let us know in the issue tracker
17:22
the repository is public, even though it's on our company, GitLab. I'm ending a little bit early. Thank you very much for your attention. Wow! You're giving some time back. That's so nice. So we have some time for questions.
17:45
So I think we have plenty of time, so please. Thank you for the presentation. You mentioned that you are publishing with some OGC APIs and I was curious to know which ones you are using. So right now we support OGC API maps and coverages in the making. Yeah.
18:08
I think we have ties at some point as well, but I think ties is like orthogonal to maps and coverages. This is great work.
18:27
Any thoughts around supporting vector data and rendering vector data? So we're using vector data like in support of our raster data, so we're mainly focused on raster data. We use it, for example, for masking out images like for the
18:41
valid areas, for example, or the footprint. We can show the footprint as well. But yeah, it's a second-grade citizen to us. We are, yeah, raster data.
19:00
You managed S3 support. Do you need to somehow pre-process the data to make it viable to pull it from S3, or how does it work right now? Well, we are using the great capabilities of GDAL. Thanks for that, Ivan. So it works best with Cloud Optimized formats, obviously, but it will also work with any other formats because this is how GDAL works.
19:24
Yeah, this is why we have to pre-process in order to like make pre-selections, use Cloud Optimized formats like GTFs. Yeah, it works best that way, but it's not a requirement.
19:49
So I guess for the actual rendering part, you're moving away from map server, right? Yes, unfortunately, yeah. It is what it is. What is the actual way that you're rendering then? Are you using like GDAL?
20:04
Yeah, we're using the Python API of GDAL. We were thinking of like using Rasterio, but like GDAL and Python API still has the best set of features, like all of the pan sharpening and the dam functionality that we also provide. It's just available through the Python API, so we're using that. Thank you.
20:39
Hello, have you considered using
20:42
OTC RP processes between Stecture and Theraviz? Yes, yes, we did. So yeah, we decided to define our own workflow formats, but we're also thinking of like standardizing the interface between Stecture and Theraviz, like this would be a great idea to use processes.
21:08
So you mentioned you are using both the Stack API and Stack Static Catalog, right? Are you using any tools or is it native implementation that you have there?
21:20
That's actually a good question, but I think we are using our own interfaces in order to access them. PyStack might be in the mix somewhere. I think we're using PyStack items as models in between the systems, but I think we're doing our own requests. I'm not too sure about that, to be honest. Thank you.