We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Deployment of AI-enhanced services in climate resilience information systems

00:00

Formal Metadata

Title
Deployment of AI-enhanced services in climate resilience information systems
Title of Series
Number of Parts
351
Author
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language
Production Year2022

Content Metadata

Subject Area
Genre
Abstract
Producing and providing useful information for climate services requires vast volumes of data to come together that further requires technical standards. Beside ordinary base processes for climate data processing like polygon subsetting, there is the special case of extreme climate events and their impacts, where scientific methods for appropriate assessments, detection or even attribution are facing high complexity for the data processing workflows. Therefore the production of climate information services requires optimal science based technical systems, named in this paper climate resilience information systems (CRIS). CRIS like the Climate Data Store (CDS) of the Copernicus Climate Change Service (C3S) are connected to distribute data archives, storing huge amounts of raw data themselves and containing processing services to transform the raw data into usable enhanced information about climate related topics. Ideally this climate information can be requested on demand and is then produced by the CRIS on request by the user. This kind of CRIS can be enhanced when scientific workflows for general climate assessment or even extreme events detection are optimized as information production service, accordingly deployed to be usable by extreme events experts to facilitate their work through a frontend. Deployment into federated data processing systems like CDS requires that scientific methods and their algorithms be wrapped up as technical services following standards of application programming interfaces (API) and, as good practice, even FAIR principles. FAIR principles means to be Findable within federated data distribution architectures, including public catalogs of well documented scientific analytical processes. Remote storage and computation resources should be operationally Accessible to all, including low bandwidth regions and closing digital gaps to ‘Leave No One Behind’. Aggreeing on standards for Data inputs, outputs, and processing API are the necessary conditions to ensure the system is Interoperable. Finally they should be built from Reusable building blocks that can be realized by modular architectures with swappable components, data provenance systems and rich metadata. General building blocks for climate resilience information systems A particular focus will be the "roocs" (Remote Operations on Climate Simulations) project, a set of tools and services to provide "data-aware" processing of ESGF (Earth System Grid Federation) and other standards-compliant climate datasets from modelling initiatives such as CMIP6 and CORDEX. One example is ‘Rook’ an implementation of the OGC Web Processing service (WPS) standard, that enables remote operations, such as spatio-temporal subsetting, on climate model data. It exposes all the operations available in the ‘daops’ library based on Xarray. Finch is a WPS-based service for remote climate index calculations, also used for the analytics of ClimateData.ca, that dynamically wraps Xclim, a Python-based high-performance distributed climate index library. Finch automatically builds catalogues of available climate indicators, fetches data using “lazy”-loading, and manages asynchronous requests with Gunicorn and Dask. Raven-WPS provides parallel web access to a dynamically-configurable ‘RAVEN’ hydrological modelling framework with numerous pre-configured hydrological models (GR4J-CN, HBV-EC, HMETS, MOHYSE) and terrain-based analyses. Coupling GeoServer-housed terrain datasets with climate datasets, RAVEN can perform analyses such as hydrological forecasting without requirements of local access to data, installation of binaries, or local computation. The EO Exploitation Platform Common Architecture (EOEPCA) describes an app-to-the-data paradigm where users select, deploy and run application workflows on remote platforms where the data resides. Following OGC Best Practices for EO Application Packages, Weaver executes workflows that chain together various applications and WPS inputs/outputs. It can also deploy near-to-data applications using Common Workflow Language (CWL) application definitions. Weaver was developed especially with climate services use cases in mind. Case of AI for extreme events investigations Here we present challenges and preliminary prototypes for services which are based on OGC API standards for processing and implementation of Artificial Intelligence (AI) solutions. We will presenting blueprints on how AI-based scientific workflows can be ingested into climate resilience information systems to enhance climate services related to extreme weather and impact events. The importance of API standards will be pointed out to ensure reliable data processing in federated spatial data infrastructures. Examples will be taken from the EU Horizon2020 Climate Intelligence project, where extreme events components could optionally be deployed in C3S. Within this project, appropriate technical services will be developed as building blocks ready to deploy into digital data infrastructures like C3S but also European Science Cloud, or the DIAS. This deployment flexibility results out of the standard compliance and FAIR principles. In particular, a service employing state-of-the-art deep learning based inpainting technology to reconstruct missing climate information of global temperature patterns will be developed. This OGC-standard based web processing service (WPS) will be used as a prototype and extended in the future to other climate variables. Developments focus on heatwaves and warm nights, extreme droughts, tropical cyclones and compound and concurrent events, including their impacts, whilst the concepts are targeting generalized opportunities to transfer any kind of scientific workflow to a technical service underpinning scientific climate service. The blueprints take into account how to chain the data processing from data search and fetch, event index definition and detection as well as identifying the drivers responsible for the intensity of the extreme event to construct storylines.
Keywords
Web serviceInformationSystem programmingModulo (jargon)TrailOpen setComputer programming1 (number)Web serviceBitInformation systemsProjective planeInformationCloud computingMechanism designTrailStaff (military)Artificial neural networkAlgorithmDifferent (Kate Ryan album)WebsiteProcess (computing)MereologyComputer animation
Thermal expansionStrategy gamePrototypeSystem programmingTranslation (relic)Software testingProduct (business)Integrated development environmentWeb serviceProcess (computing)DisintegrationFeasibility studyScale (map)Level (video gaming)Level (video gaming)Software testingDifferent (Kate Ryan album)Term (mathematics)TestbedProjective planeCodeFlow separationComputer programmingComputer animation
MathematicsWeb serviceLattice (order)Line (geometry)Domain nameDialectDifferent (Kate Ryan album)Computer animation
Time domainInternet forumInformationGroup actionFocus (optics)Pressure volume diagramDisintegrationVisualization (computer graphics)Variable (mathematics)Workstation <Musikinstrument>Reduction of orderData fusionStatisticsMathematicsProcess (computing)Function (mathematics)Artificial intelligenceProduct (business)Mathematical analysisDecision theoryWeb serviceInformationWeb serviceGroup actionLattice (order)Domain nameVariety (linguistics)System programmingLevel (video gaming)Information systemsIdentifiabilityComputer animation
InformationWeb serviceSoftware developerDecision theorySatelliteAsynchronous Transfer ModeNumerical analysisReduction of orderVolumeIntelInternetworkingStatisticsComputer networkArtificial neural networkSpacetimeContext awarenessContrast (vision)System programmingTerm (mathematics)Cubic graphComputer animationProgram flowchart
Client (computing)Web serviceOpen setOpen sourceTerm (mathematics)DampingMechanism designData storage deviceSelf-organizationPolygonCuboidSet (mathematics)CollaborationismWeb serviceSubsetClosed setComputer animation
Web serviceSkeleton (computer programming)Template (C++)HTTP cookieInstallation artProcess (computing)World Wide Web ConsortiumExact sequenceParameter (computer programming)Point cloudInternet service provideroutputMereologyComputer programmingProcess (computing)Scheduling (computing)PrototypeDomain nameAlgorithmTemplate (C++)HTTP cookieRepository (publishing)Line (geometry)Web serviceBitThermodynamischer ProzessGroup actionInternationalization and localizationProjective planeEvent horizonInformation systemsExtreme programmingWeb 2.0Skeleton (computer programming)CodeVapor barrierWordMultiplication signFrame problemHeat waveComputer animation
PrototypeProcess (computing)MathematicsDevice driverInformationHeat waveVirtual machineExtreme programmingEvent horizonComponent-based software engineeringCodeVulnerability (computing)Price indexWaveWeb servicePlastikkarteGraphical user interfaceSystem programmingSoftwareSelf-organizationDemo (music)Software frameworkSoftware testingDenial-of-service attackHeat waveSet (mathematics)PrototypeAlgorithmThermodynamischer ProzessWeb serviceEvent horizonPredictabilityVirtual machineLine (geometry)Flow separationProcess (computing)Data storage deviceResultantDialectUniform resource locatorWave packetDomain namePattern recognitionVulnerability (computing)Digital photographyDampingMultiplicationMultiplication signMoment (mathematics)Device driverType theoryWeb 2.0Raw image formatSource codeSubject indexingComputer animation
Time domainGroup actionProduct (business)Set (mathematics)Internet forumComputer animation
Transcript: English(auto-generated)
Yeah, welcome everybody and thanks a lot to everybody who made the way up until this hidden corner on the academic track. Yeah, the following talk is about the deployment or deployment mechanisms of artificial
intelligent enhanced services for climate resilience information systems. So basically the questions of how to bring scientific algorithms into cloud computing services and so that the end users can have an easy access to information on demand. That's and there for sure whether it's filled out with OGC
standards because yeah, I'm a staff in OGC. OGC has two big parts, big programs. So one is the standard standardization program and we had been hearing that in many different talks about the OGC standards and that then but then on the other side
there's also the innovation program where we are involved in quite a lot of different projects and the ones which is a bit related to climate are here, but feel free to check out the website. So there are plenty of other projects as well. And that's where we are fiddling around with
technical challenges and trying to find good recipes and if we are making a good job, then maybe they're ingesting into their standardization as well. And then also while the OGC is or in the OGC innovation program, we are working on quite different levels of
technical majority levels and so from very basic concepts up to then the testing and making the interoperability experiments and and then also well in terms of projects and we have small projects, little coding sprints for a couple of days or then also well the test beds
which are running over a couple of months or then long projects for several years. So that's just kind of showing the flexibility that we can do and and now if you are focusing on climate that's something what we are pitching up currently as a new road in OGC and with a new domain that we are
focusing on and last year and that we had kind of a lot of different ad hoc sessions, climate sessions during the OGC member meetings as well as then also dedicated or in line with the United Nations
COP26 or the the regional climate weeks in Africa and Asia Pacific, etc. And then colleagues from ECMWR, for example from ESA, from NASA, from from the Canadian colleagues, Australia, etc. They were
contributing to these sessions and then we had a very nice exchange and building up a community and about the questions on how to identify principles, so findable, accessible, interoperable, reusable
aspects in the climate domain. Not only for the data, but then also for the entire system itself. So how should climate information, climate resilience information systems be be designed so that we can have a better synergy and that we can have kind of a good exchange.
That's what we are talking about here in the Phosphogy very often and out of this all these discussions we have been setting up a climate resilience domain working group where you're all welcome to join and continue with the discussion and then the next working group meeting will be in the 5th of October during the OGC member meeting in Singapore
where we are electing the chairs and yeah, you're more than welcome to join. What we are talking about here is always the question that in climate or in climate related domain we have a huge amount of data, which you can see here on the left side and
the question is always how to concentrate that, how to merge that down, how to fusion the data, how to bring that from the raw data into a sensible information that is then useful for the scientists or policymakers, for non-technical persons, for a variety of very
homogeneous, a very heterogeneous user group. That's summarized in climate service or science service if you are including all different domains as well. On a technical level and here comes the the question of
interoperability and the fair principles. What we try to do especially then in the context of OGC is building up the blocks in the way so that they are fitting to each other quite easily. So that we then if you would like to set up your own system
wherever you are and however the system should look like and that all the blocks which are there are fitting together and you can build it up and in kind of in a cubic building block system. So that's the entire idea of all the standardization.
So why we are sticking to standards or why it makes sense to stick to standards in in contrast of building up your own individual ideas. So that's what we are here. And then in terms of the climate or climate related
building blocks, things that we're not starting from scratch, but there is quite a lot of things already out there. What I'm showing here is all open source. You can find it in the in a GitHub organization called Birdhouse, but then left and right we have other things also laying around.
There are two things already implemented in ECMWF C3S. It's the Rooks and then also the deployment mechanisms with Ansible Doggers and with Conda. Rooks is a service for polygon subsetting, bounding box subsetting. So that's already implemented there and we are in close collaboration with ECMWF and talking about how to bring the other blocks
into the climate data store as well. But yeah, feel free to to take that if you have a need on it and I'm showing you now a few other things what we have been
developed and which is upcoming where you can see that. And the idea is also well if it is working fine, so if we are kind of sticking to the standards, if you're sticking to OGC standards or what standard whatever, then it's possible to kind of exchange and and have the
synergies in between of the different services. And then during the or within the domain working group focusing on the United Nations policy frames, that is always the questions of how to enable the synergies in between of the different countries and the services which are around.
One of the things that we have, so now it starts to become a bit more academic and more technical, is to help out and building up these blocks yourself. So if you have a good algorithms that you would like to wrap up and then maybe provide that for C3S or for whatever service you are focusing on, then we have a template which is called the cookie cutter.
You find that also in the birdhouse repository. You just need to install that. So a couple of letters, a couple of words, a small command and then you have it in place and by running the cookie cutter
it's asking you a couple of questions, takes not so much time, some minutes, and you have the skeleton ready. And the skeleton is then in line with all the OGC standards. So it is respecting all the required yeah, all the all the required
standardizations. And then comes the the big work that you need to do is that fill it with your algorithms and kind of the code that you have needs to be set in place. And then also here you need to respect how to pass through the different arguments that it is then fully automatically running on on the cloud.
But once this has been done, once you have ingested that as well, then it's ready to implement and deploy that in C3S for example or wherever you you want to have that. So that's kind of here. We were working on that to lower the barrier so that the community can grow and then everybody who wants to
provide services in international or local climate resilience information systems. That's the way how you can do that or that's the way we are proposing. One of the things, this is a bigger project where OGC is involved together with a consortium, quite scientific
focused consortium. This is a European project CLINT. It's called CLINT climate intelligence. Here focusing on artificial intelligence to have a better understanding of extreme events and extreme event detection. So what we're doing here with the scientific colleagues is they are providing
the algorithms, the scientific algorithms and how to detect heat waves, tropical cyclones and etc. Things like that. And focusing on
seasonal forecasts so that's the dedicated these are the input data where we are focusing on. And then the part of OGC innovation program is that we are wrapping up these algorithms into OGC API processors or formally that was web processing service and
in the way I was showing before and having a building block and then with the idea that by the end of the project is going to be ingested or is going to be provided for the Copernicus
services. It's provided they have to do the deployment on their side then. But we are far ahead on the schedule. So the first processes, the first prototypes are already in place. So we have plenty of time to think about how to fiddle around and how to bring that into the climate data store and also where currently climate data store is being rebuilt and that we can think about how to
bring, yeah, how to deploy that in an easier way. So the first prototype that we have in place, which is up and running, there is a
the URL that Clint Dicker said, the German Climate Computing Center is hosting that for the moment. If you would like to play around on that, it's just type in this URL and what the process is doing is based on artificial intelligence.
So in the middle you have a data set with quite a lot of missing values. So basically everything is missing. And then on the left side you have a training data set. So the ground truth and where the artificial intelligence is being trained on and
kind of learning how the values should look like. And then on the right side you see the result. And the results are quite promising and quite astonishing. We took an example here in Africa because of, well, Clint is basically focusing on Europe,
but there were not so much data sets with too many missing values. So we had to take a data set from Africa. And currently we are fiddling around also with other sources, so radar data, and this is being possible as well. And the algorithms behind is coming from
your face recognition algorithms. So when you have a photo and there is something missing in the picture or there are scratches inside so that the algorithms could be trained and fill in the rest of the face.
So that's the the scientific algorithms behind. But if you have more questions on that then my colleagues from Deceresat can tell you all the details about that. Then another one is more on is we are facing here the the challenge of data pipelines and that is
heat waves and warm nights. And what we are doing is are several steps then. So there were processing services with multiple steps. And the first is a detection of the event, so the definition of the index, which has been detected by machine learning algorithms. And then once the
event is being detected and that we're piping through quite a lot of data and that making detection of the drivers to understand why this extreme event has been occurred. So if you have a heat wave, why is the heat wave occurring? So it is
depending on, I don't know, soil moisture or the westerlies or what is the reason for this event. And then that's been then detected by as much data as you can ingest there. And once this has been done
you can make the assessment of the drivers and do the prediction for the future. These are then the story lines which are coming out. We have several test regions because machine learning you need to reduce.
You can do that on a larger domain. So we have some test regions in Europe as well as in Africa and try to understand better how good that works. Okay, and to understand, well, so what we are promising for for the future that is missing values is already
very is already in place. You can take it and improve it on your side as well. Heat waves, we are on the way of having that as a prototype up and running quite soon. And then we are also focusing on hydrological floods. We'll
using artificial intelligence and then drought vulnerability and tropical cyclones, which is coming then from ECMWF itself. And I think with that, oh no, yeah, and for sure we have documentation in place. So if you would like to do it yourself, if you want to have a known
climate building block on your side, if you would like to contribute to that, then here is the documentation. It is the very draft and very raw beginning. So if there is something missing, don't hesitate to reach out and then we are
completing that. It will start to grow in the future. But you are more than welcome to contribute. And with that, I would like to thank you and I'm looking forward for the discussion.