ZOO-Project: from OGC WPS to OGC API - Processes Part 1 and Part 2
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 156 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/68577 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
FOSS4G Europe 2024 Tartu156 / 156
6
33
35
53
55
59
61
67
70
87
97
99
102
103
104
105
107
111
121
122
123
124
125
126
127
128
134
144
150
151
155
00:00
Standard deviationCore dumpOpen sourceComputing platformSoftware frameworkChainingWeb serviceImplementationProcess (computing)Generic programmingCodeKernel (computing)Revision controlOpen setMereologyJava appletJava appletMereologyMessage passingRevision controlOpen sourceKernel (computing)Web serviceComputing platformData managementThermodynamischer ProzessFraunhofer-Institut für Informations- und DatenverarbeitungQuery languageProjective planeServer (computing)SoftwareClient (computing)Core dumpTestbedCodeSource codeInteractive televisionMathematical singularityVirtual machineLatent heatFrequencyArithmetic meanFunction (mathematics)Level (video gaming)ChainLogicProgramming languageStudent's t-testLibrary (computing)BitImplementationExploit (computer security)Web 2.0GeometryFormal languageMultiplication signComputer architectureComputer fileCASE <Informatik>Cartesian coordinate systemDependent and independent variablesState observerStandard deviationContent (media)Medical imagingTask (computing)Library catalogRemote procedure callModule (mathematics)Internet service providerPhysical systemCuboidLine (geometry)Landing pageSocial classOpen setDifferent (Kate Ryan album)Web pageConformal mapSinc functionComputer programmingLecture/ConferenceComputer animation
09:18
Generic programmingThermodynamischer ProzessPoint cloudArchitectureComputing platformComputer-generated imagerySoftware testingDependent and independent variablesRevision controloutputOpen setThermodynamischer ProzessInteractive televisionWeb serviceProjective planeLibrary catalogEndliche ModelltheorieMereologyComputing platformExtension (kinesiology)Software developerImplementationDescriptive statisticsServer (computing)Object (grammar)Conservation lawData storage deviceCodeStack (abstract data type)Concurrency (computer science)SoftwareInterface (computing)Mathematical optimizationRevision controlPerformance appraisalFile formatSuite (music)Latent heatGraph (mathematics)InferenceUser interfaceArithmetic meanPhase transitionLevel (video gaming)MetadataComputer fileReverse engineeringLine (geometry)Computer programmingDatabaseCartesian coordinate systemDemosceneHeat transferTask (computing)Virtual machineLink (knot theory)Multiplication signInformationClient (computing)Source codeCombinational logicStandard deviationKernel (computing)Core dumpMedical imagingArtificial neural networkWeb browserCellular automatonWebsiteInstance (computer science)Group actionAlgorithmFile viewerSelf-organizationShared memoryFreewareMessage passingTestbedLibrary (computing)Programming languageSupercomputerOpen sourceState observerFormal languageDiallyl disulfideCubeSlide ruleCASE <Informatik>Shape (magazine)Computer animationLecture/Conference
18:33
Computer-assisted translationThermodynamischer ProzessComputer animation
Transcript: English(auto-generated)
00:00
Hello, everyone. I am Gérald Fenno from Geolabs company, a very small SME, created back in 2006. And I am glad to be accepted by the conference committee to be able to present the ZOO project and show you how we moved from the OGC API, OGC WPS, sorry, web processing services, to OGC API processes part
00:23
one core, and also the part two draft specification, which is about deploy, replace, and undeploy. So the ZOO project is an open source platform implemented in C and C++ mainly. It's released under MIT X11 license.
00:42
And it implements WPS standard one, version one, version two, OGC API processes part one core standard, OGC API processes part two, deploy, replace, and undeploy draft specification, which is ongoing, and also part of the OGC API processes part three workflow draft specification, especially the conformance class named
01:03
remote processes, which lets you build complex query involving multiple processes that can be running on your own server or on a remote server. So the initial idea was to use all the open source software libraries in a free only way, in an easy way,
01:23
in a standard way. So this is why we invented the ZOO project back in 2008 during the FOS4G in Cape Town. The idea came there. And then since Sydney, we are trying to present every year at the international FOS4G when it was happening, not during the COVID period,
01:42
obviously. And we try to make a release for every new FOS4G. Also, back in February last year, ZOO project become an OGC reference implementation for both WPS2 and OGC API processes part one core.
02:01
And also, ZOO project graduated last year as an OSGO project. So we are willing to thank our mentors, Franco-Merdum, which was the initial one, Dimitris Kotsinos, which take the lead afterward, and finally, Tom Kralidis, sorry, which made us realize our dream to become an OSGO project.
02:23
So finally, in March 2024, ZOO project 2.0 has been released. And I will show you some of its capabilities. So this is a PSC of the ZOO project. I would like to thank Fabrice Brito from Terra Do E Italy, which has joined lately the ZOO project PSC.
02:45
These are a few of the committers, but you can directly go on GitHub to review this. And this is an overview of the open WPS platform as we named it because we started, as I told you, with WPS. So we have the ZOO kernel, which is a WPS server which is responsible to interact
03:03
with your service and execute with the client, obviously, and execute your services. We have the ZOO API, which is a JavaScript API which is running on the server side and which gives you the opportunity to create a chain of processes, including logic inside the chain. And then we have the WPS client,
03:20
which is a bit obsolete nowadays because we are not using WPS anymore. At least we are using OGC API processes in many places. So obviously, it is working on every single platform. Actually, we are supporting part one and part two since version 1.9. What is interesting with the ZOO kernel,
03:42
which is the engine which is taking the responsibility to handle your service execution, is that it can interact with almost any programming languages from C, C++, Fortran, Java, PHP, Perl, Ruby, Python, C Sharp, and JavaScript. At this step, when we arrived to interact
04:03
with all these programming languages, we were thinking it was a great achievement. But yet, we needed to maintain the source code of these services. So it was still some work. So finally, we decided to integrate with existing processing applications, which
04:21
are providing catalogs, such as Orfeo Toolbox and Saga GIS, making all their application or remote module for Orfeo Toolbox available as WPS or OGC API processes without the need to write one line of code. So also, since the beginning, we integrated with Map Server,
04:43
the C library, for being able, in case your service is outputting any GIS-related things, then you will be able to automatically get them published as WMS, WFS, or WCS. So this is an example I talked about since a long time.
05:02
Also, since WPS 1, we have the get status service, which has been renamed as a request in WPS 2, which is named get status, and which is the same, which is available in OGC API processes, part 1 core. Also, back in 2017, we were involved
05:22
in the Geosuit project led by IARD in a French research laboratory, where the project was used to execute remote tasks. Actually, it was only Orfeo Toolbox applications. Remotely, on an HPC server. Then we have implemented the OGC API processes, part 1 core.
05:44
So in the OGC API, every OGC API, as Joanna was mentioning earlier, you have a landing page, and you have also the capability to request the content as JSON or as HTML. So we took this as an advantage
06:01
to implement some things that we are doing also in MapMint for years now, since 2010, which is reusing a Cheetah templating system to provide HTML pages based on the JSON provided by the OGC API processes server. As Joanna said, we became a reference implementation
06:20
last year for both WPS 2 and OGC API processes, part 1 core. We have integrated work which has been done late in 2015, but it was integrated only a few years back. I cannot recall exactly the date. Where we are now having two different kind of ZooCanal.
06:41
We have the main ZooCanal, which is taking the request. Let's say the front ZooCanal. And then we have the ZooCanal fast process manager, which is responsible to grab messages, so asynchronous request execution, to interpret them
07:00
and handle them directly, potentially, on another machine. During the testbed 19 HPGC activity, the Zoo project has been modified to actually reuse part 2 draft specification and being able to automatically deploy singularity container on HPC and then execute them using the OGC API processes,
07:23
part 1 core. Also, back in 2022, during OSGO, OGC ISF Apache Software Foundation Code Sprint, in which we participated, we integrated with Map Server 8, meaning that rather than only supporting WMS, WFS, and WCS,
07:44
we are now also providing automatic output of OGC API features, at least at the level supported by Map Server. We are willing to thank Microsoft and Google. So Microsoft Azure for open source, which is a project where you can apply to have a grant to be able to use the Azure platform almost for free,
08:04
meaning that they will give you a given amount of money, and you can use it for your own purposes. We were using it, actually, to deploy Zoo project DRU, which I will speak about later, to deploy it on Azure. And also, since 2015, with Raja Chinde,
08:22
we started to put Zoo project in a Google Summer of Code and propose a project to students that are paid by Google to participate in the community. And since then, every year, we have a new student which can participate either in Zoo project or MapMint.
08:41
Also, the Zoo project is now included in the Eoepka project, Earth Observation Exploitation Platform Common Architecture. And it is used there as a processing engine, obviously, which is able to deploy a CWL file, so Common Workflow Language, where you can embed a bus,
09:01
your Docker image you are willing to run on, potentially your source code, even, or the application you want to call. You can deploy this as a Node-GC API processes part one core standard, and then interact with it. So we are relying on a best practice for Earth Observation Exploitation package, which defines that stack should be used for both input
09:23
and output. This is what you can see on the slide, the stack staging and stage out step, which is involved in every execution. So actually, this is an overview of what is CWL, meaning that you can have your application implemented
09:41
in any programming language. Then you will publish a Docker image that you will reference from your CWL file. You deploy this CWL file on the ZOO project, and you will get a standard OGC API processes that you can interact with. This is the most important part of this support for the deploy,
10:02
replace, and un-deploy. It is a ZOO calibration runner, which makes your task able to run directly on a Kubernetes cluster. And obviously, if the task can be run in concurrency or in parallel, then you will have a pod that will pop automatically to handle the current step.
10:22
Also, during the testbed 19 GeodataCube activity, in the ZOO kernel, we integrated the filter in and filter out capability, which means that traditionally, the ZOO kernel is handling the request as it comes. But nowadays, before handling of the request,
10:40
you can implement a filter in, which can do almost whatever you want. And also, you can have a filter out. You can have actually as many services in vocation in both filter in and filter out. So you can add more capability to the current implementation without having to implement much things, but simply relying on existing capabilities
11:03
from other software. So we started this filter in and filter out by showing that we were able to deploy OpenEO Graph as first citizen in OGC API processes part one core using the part two draft specification. Also, we integrated the ZOO project with EO API
11:22
simply by using filter in or filter out to add the capabilities. And this way, we can have the data nearby processing or reverse sense. Also, during the testbed 19 machine learning transfer activity, the ZOO project was used to use in combination
11:40
with NVIDIA, Triton, inference server, and the Notron Viewer. You are relying on the Open Neural Network or NNX format to store your model. So we are providing some kind of model catalog where you can list the model, access one model description. When you have this model description, it come in JSON.
12:01
But if you want to visualize it, you can use the Notron interface transparently. It comes in a single open API. And then you have inference as a service too. But at that time, we did not investigate if we were able to do that with CWL. So since then, we are involved
12:22
in climate and disaster resilience pilot first phase 2024, in which we are finally, we have demonstrated that we are able to implement the inference directly within CWL language. Also, the project is involved
12:41
in the OGC Open Science Persistent Demonstrator when we are willing to share algorithms amongst organization and make them fair. Actually, here you can see the Galaxy project in action where you can build a workflow. There is a 52N company which is responsible
13:02
to create an OGC API Processes client, which is able to interact with any OGC API Processes server instance. And in this case, you can see a workflow where we are interacting with some processes coming from CRIM, from Canada,
13:20
and some other processes coming from GeoLabs. About the ongoing development and the goal we are willing to achieve before the end of this year, we are willing to get the OGC API Processes part two, which I think it's in very good shape, to be released and validated by the OAB.
13:42
Then we are willing to investigate the proposed extension to OGC API Processes, which is a pass-through workflow, which is, to me, containing too many things at this step. Maybe we have to investigate or split it in multiple parts. We are also willing to integrate the support
14:01
for CWLprov to conserve provenance and store the workflow run as a research object using linked data. And we are also willing to evaluate Clued Optimized format when they are using for processing data on HPC. So then we have the ZOO services.
14:20
You remember I was telling you that we have the ZOO kernel, ZOO service, ZOO API, and ZOO client. So the ZOO services is basically a growing suite of services where you have the source code and the metadata information, which are separated. I don't think it is the same in every implementation, and the metadata can be also stored
14:41
inside a PostgreSQL database. So here you have a very simple example of a yellow world, and we, as Einstein said, we try to keep things simple, stupid, and kiss. These are the engine we were supporting. So we have more than 700 OGC API processes, or WPS services,
15:00
because actually when you are moving from WPS, at least using the ZOO project, if you are moving from WPS to OGC API processes, you don't have to modify one line of code in your service source code. So this is the ZOO API, which is commonly used even nowadays, even when we are relying and interacting with other OGC API processes,
15:21
even if it's only supporting WPS1 behind the scenes. With this, we have built up MapMint version one long time ago, back in 2010, and we were using, obviously, the amazing GEDL tool to produce relevant information for our clients. Then we have the ZOO client, which is this time a JavaScript library,
15:42
but which make your client application so your browser able to communicate with the WPS server. Thanks to this, we have made the example available on the ZOO project website for SAGA GIS, our third toolbox. Then we have made MapMint to client,
16:01
which is currently still in cell. So we have moved to GitHub. We are currently, for every push you make, we are building the OGC ETS, and we are running it through the new code just to ensure that we are still conformant with the standard.
16:22
You can find ZOO project on Docker Hub as an open-source software. So you can build and deploy a ZOO project easily on your own platform. You can also deploy the ZOO project DRU, which is the one involved in a EO EPCA project to deploy it on your Kubernetes cluster.
16:41
So you can go on Artifact Hub and download the M-chart and follow the documentation on how to use it. And last but not least, we have built up on top of this ZOO project an SDI, which is named MapMint, which is a user interface for Map Server.
17:00
So eta, if there is any question, I will be, I hope I will be able to answer them. Thank you very much, Sherald. So who wants to ask the first question?
17:37
Thanks, Sherald. I would like to know, because in both part three of OGC API processes
17:43
and also in the GU DataCube API, there's mentioned Open EO. I'd like to know what's the relationship with Open EO. Is there any interaction planned or? Actually, I'm not sure. We are trying at the OGC to find a link
18:01
between Open EO and OGC API processes. This is why actually I was demonstrating this publication of an Open EO graph and using it after a while as an OGC API processes part one core. But as Matthias would have said, I think Open EO is already coming with a lot of tooling around it.
18:22
So maybe we should do the reverse, meaning that we probably should adapt the OGC API processes to be more compliant with Open EO rather than making Open EO integrated within OGC API processes. Thanks. Thank you, Fabien.