Building cloud environments with open source software to offer processing of large environmental data sets
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 295 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/43571 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
FOSS4G Bucharest 201913 / 295
15
20
28
32
37
38
39
40
41
42
43
44
46
48
52
54
57
69
72
75
83
85
87
88
101
103
105
106
108
111
114
119
122
123
126
129
130
131
132
137
139
140
141
142
143
144
147
148
149
155
157
159
163
166
170
171
179
189
191
192
193
194
195
196
197
202
207
212
213
214
215
216
231
235
251
252
263
287
00:00
Open sourceProgrammer (hardware)Software frameworkHorizonProcess (computing)SoftwareIntegrated development environmentSet (mathematics)Point cloudBuildingFatou-MengeCoefficient of determinationComputer animation
00:22
SoftwareProcess (computing)Integrated development environmentSet (mathematics)Gamma functionOpen sourceBuildingPoint cloudService (economics)Module (mathematics)Range (statistics)Data managementMathematicsStaff (military)Operations researchEstimationLevel (video gaming)Physical systemVisualization (computer graphics)Musical ensemblePublic domainFile formatLatent heatRankingVolumeLibrary (computing)Parameter (computer programming)Computer configurationUniverse (mathematics)Service (economics)Staff (military)SoftwareWeb portalMultiplication signParameter (computer programming)Web serviceState of matterMixed realitySelf-organizationLibrary (computing)Single-precision floating-point formatGraph coloringImage resolutionPairwise comparisonMappingMusical ensembleProcess (computing)Endliche ModelltheorieOperator (mathematics)DialectProduct (business)Codierung <Programmierung>MetrologieVisualization (computer graphics)Web 2.0Workstation <Musikinstrument>Open sourceSet (mathematics)Order (biology)Physical systemFile archiverLaptopData centerBridging (networking)ResultantReal-time operating systemMetreMereologyProjective planeObservational studyExtension (kinesiology)Series (mathematics)Reading (process)Sinc functionHypermediaSimulationOpen set19 (number)Sampling (statistics)Computer animation
06:11
Open setService (economics)State of matterParameter (computer programming)Visualization (computer graphics)Active contour modelAreaRule of inferenceKey (cryptography)EstimationRepresentation (politics)Scaling (geometry)Directory servicePort scannerSystem callComputer-generated imageryVolumenvisualisierungSanitary sewerFiber (mathematics)Client (computing)SharewareImplementationType theoryBuildingAxonometric projectionCubeComputer networkHuman migrationSoftwareAlgorithmSupercomputerProcess (computing)Data modelHill differential equationSimulationLevel (video gaming)Link (knot theory)Local ringLibrary (computing)Alpha (investment)Phase transitionDirectory serviceWeb browserMachine visionFile formatHorizonCodeGraph coloringWeightDifferent (Kate Ryan album)Arithmetic meanProjective planeService (economics)BuildingCumulative distribution functionMetadataMultiplication signServer (computing)Client (computing)MereologyParameter (computer programming)Functional (mathematics)Universe (mathematics)State of matterBasis <Mathematik>Field (computer science)BitHuman migrationPresentation of a groupTimestampMetreSeries (mathematics)Rule of inferenceGroup actionQuantum stateMessage passingPort scannerComputer simulationGoodness of fitPoint (geometry)Military baseComputer animation
11:29
Field (computer science)Mechanism designArc (geometry)Point cloudComputer hardwareData modelServer (computing)Sanitary sewerSet (mathematics)Interface (computing)Observational studyComponent-based software engineeringDisintegrationRule of inferenceVisualization (computer graphics)Stack (abstract data type)Software developerIntegrated development environmentProcess (computing)Point cloudResultantSimulationHorizonProjective planeVisualization (computer graphics)Message passingComponent-based software engineeringCartesian coordinate systemDatabaseComputer simulationPlanningDemosceneService (economics)Scientific modellingFile archiverWeb portalMultiplication signState of matterBuildingEndliche ModelltheorieSatelliteSupercomputerField (computer science)SoftwareMedical imagingOpen setReplication (computing)Rule of inferencePoint (geometry)Data storage deviceDataflowFrequencyView (database)Similarity (geometry)Computer animation
16:47
Software developerStack (abstract data type)Integrated development environmentHorizon1 (number)Product (business)Projective planePredictabilityWeightFlow separationComputer animationMeeting/Interview
Transcript: English(auto-generated)
00:07
Thank you Hello I'm Milena. I'm originally from Serbia but now I work in the UK in European Center for medium-range weather forecast and I will present our efforts to make our
00:21
Data as you can could see yesterday in Julia talks. Our data is Somewhat complicated for users that are outside of meteorological community. So I will present our efforts to bridge this gap So we are European Center for medium-range weather forecast we are based in Reading we are established in the
00:42
70s and since 1979 we are producing weather forecast. I mean we are inter International organization with 34 member and cooperating states and we have around 350 staff members and growing we are both operational service and research center and we
01:02
Operate to compare Nikos services Copernicus climate change service and Copernicus atmospheric monitoring service We are now in the UK, but we are moving our data center to Bologna next year, so we will be in two countries
01:24
Our operational Activities are Twice twice a day we produce High-resolution and ensemble weather forecast and another twice a day. We produce global model that is
01:41
Which data is used as? entry for our member states regional models on high resolution We produced hundreds of terabytes of data. And we also have this data available as web maps as hundred in hundreds of Thousands of web maps updated twice a day and we
02:01
Disiminate all our data in real time twice a day to 200 destination worldwide We are also producing everyday research data It is also hundreds hundreds of experiments from research at our center and in our member states and we also producing
02:20
Forecast and climate reanalysis it all adds up to our big metrological archive which is now over 300 petabytes of data and 250 petabytes added every day all of this doesn't make our data very
02:42
easy to access even though we do have very sophisticated dissemination system that is Disiminating all these terabytes of data to our users twice a day. We have web services. Our data are available through Our web portal and through
03:01
WMS non users can actually Process all of our data and lots of our ensemble data sits unused Because it's a lot to download and then to process so we are looking into better ways to
03:24
Bridge This gap between users and data Which presents our first challenge How do we actually get more users and more non-metrological users to use our data?
03:41
Another challenge that we have is that users want to Get to know our data they want to visualize their data They want to easily share results of their metrological work to interactively work with our data But they don't want to spend a lot of time learning another new Visualization library that works very well with metrological data
04:02
To spend a lot of time defining the colors for each and every metrological parameter and we have over 200 200 of them which leads us to Then having something like this for every single parameter or something like this when you just visualize you get every everything is the same
04:21
So another challenge is how to make this visualization process of so many metrological parameter easier for the users We can help because we have already Over 30 years of experience of working with metrological data Visualizing metrological data we have our in-house
04:42
Software that is Now even open source We have magics that is putting library. It is using our own also open source easy code library to decode metrological data and for example approach for for productions on top of magic sits Matthew, which is a metrological workstation to
05:04
visualize and process metrological data and also we have easy charge, which is that portal to To for forecasters to actually use metrological data We want to use our experience with all this to offer automatic styling to offer
05:22
Easier work with metrological data with Jupyter notebooks and to offer some kind of light WMS service that will make Users work with our data much more easily easier So we start meet with magics magics is Over 30 year old
05:42
Plotting library it is it knows our data very well But to actually find how you visualize your data You need a lot of parameters over the years magics for just plotting It has over a hundred parameters that you need to know in order to set your visualization
06:01
Well, but user don't want that user just want I give you this parameter and tell you don't give me automatic settings I want to see how it looks like So We did a little project projects we have we had Over 250 parameters already visualized in easy charts, and we wanted all of these styles
06:26
somehow imported into magics to have It available for users on demand so It was our basis If you want to know more about easy charts, there will be a presentation at the 11 in the Rhapsody a ballroom
06:50
So these these parameters in these styles are very well known by our member states are forecasters and Lots of time they they want okay. I have your data. I want to see it as I see it on easy charts
07:02
I don't want to invent. I don't want to go through some code and Define colors for every single parameter So what we did with Magics already knows methodological data and it reads
07:22
Metadata, and then we created rules for for each parameters that is recognized Okay, we have this parameter applied this style to this this parameter and then automatically it Applies the style
07:41
Grib is our methodological Format but net CDF is another more widely used format, and we want to have a We we implemented also support for net CDF, but x-ray is
08:03
Library that is now most more and more present and it it has a support for both grip and both net CDF So we want also to have support the same support that we have for a grip and that's the f we want to have it for x-ray because
08:21
That's the future so All this styling is used in our What we call skinny WMS, so it's a lightweight WMS Service so what it does it scan if you have a directory with your Metrological data it and you don't know how to work with this metrological data. It's can you use it scans the directory
08:48
Picks up metadata and then tells magics. Okay, you have this metadata. Please find me the style and get me the map so This service returns three functions we get a bit build small get capabilities get map and get legend
09:03
So you can use these get capabilities to any? WMS Client but it also has starts a small Flask server and if you want to just inspect your data, you can put this local host Link in your browser and you will get your data visualized
09:24
Right there So this is just an embryonic work. It's it is just in alpha phase started so we need there's a even though we've imported all the Recognized parameters have it all of their styles. There's a lot more to do
09:44
meaning we want to add more styles and support for more for time steps and for more parameters and the support improve support for projection How to use it well you just download it with people quanda and then start with skinny WMS and give it part path to
10:03
your data and then put the Link in your browser or put it in your WMS client and you have your all of your data in this directory visualize to inspection What does it have to do with cloud? So? Gives it brings us back to our first challenge
10:22
We have all this data we disseminate it but it's For non met service users in can be challenging just to download this data and then to process it So what we want to do is we want to bring The data the users to the data not the data to the users and we also want to bring
10:42
users to the processing and the data We Because of this reason ECMWF is joining another couple of private companies universities and research centers in Hidalgo project, which is a European
11:02
Horizon 2020 project that Has a vision of solving some of the global challenges by uniting scientists from different fields working together on solving some of world challenges the challenges that we are working on our
11:21
simulation of migration of refugees Simulation of air pollution on city level and the social Simulation of messages on social networks ECMWF is involved in the first two Simulations because they want to use our data, but they don't have absolutely any experience in using meteorological data
11:46
So we are helping them Working with our data and getting to know with it and Processing it, how do we see Hidalgo? So we have this
12:02
like a building workflow from HPC through cloud through visualization service to end-user so end-user doesn't know anything about HPC Or how to work methodical data, they just want to run there there may be NGO who is wants to see where
12:21
refugees will move in the next In the future and they they just start Working in Hidalgo project. They run the simulation and everything is Done through orchestrator behind the scene so We are using the experience in this project to move from this
12:44
How it's done today So we have we run our model we have Fields in the database and then we either push it to the user or have a user pull it from the archive Well, what we want is to have both users and service together and
13:02
And users together in the cloud so data doesn't leave our center if the user just wants one data and If they have they will have Data and software on the on the cloud they will if it's a
13:21
Advanced user who wants to run their model and simulation. They will have our infrastructure to To run their their simulation, but already they will have data too, so they don't have to move any data We are partnering with You met such together because our data is
13:41
Connected we have the same users and both of our data is Too big to actually move it and processing them together So we we already have some experience with the building a cloud
14:03
So we were involved we are involved through Copernicus to building Copernicus climate data store, which is some something similar but with all the data open so it has Climate data climate reanalysis also some satellite data and river discharge data and
14:23
Users can download data and process it but they can also down processing using CDS toolbox and they can do build a their application or process the data directly on Copernicus cloud And some of the applications are already using our small skinny WMS to visualize their
14:45
Results of their work one of This this image is showing the one of the first application that is that is actually so they they are building application for climate sustainability Sustainability for tourism and they are calculating touristic indices and visualize it using skinny WMS and magics
15:09
So Bring bringing the user and the data together and enabling Easy visualization are very important ways of getting more
15:22
users outside of our little methodological community and getting our data used more so we we Have plans to create more rules for automatic styling to Enable users to create their their own Styles to end
15:40
To do more work on this skinny WMS and that the time component and to improve actually magics by using it in reasonable data portals CDS toolbox and more horizon 2020 projects so We are building a cloud federated cloud together with human sat to facilitate this and then we started this year
16:06
Two year pilot study That will have our member state having their applications on this cloud and also horizon 2020 applications working on this cloud and Then we want to integrate this skinny WMS on the cloud through Jupiter and I pi leaflet
16:25
So it's used by even more even bigger community so this is Or Enabled by this
16:40
Horizon 2020 project and we also have a we are also hiring Thank you, thank you so questions there's many people so expect many questions
17:20
Which
17:23
It it reads grip and net CDF data, which is methodological data, but it could be adjusted it could be built Extended to support anything else and build WMS I
17:46
Have a quick question while others think is I don't work with Meteorological data, but I was contacted by another European project called primavera I don't know if you heard it. Do you have any contacts with them? Maybe I'm completely wrong and I don't it's it's still they work on meteorological data and they do some predictions
18:07
On several European projects, but I think we could cooperate do yes Other questions
18:22
Going once Last chance. Okay. Thank you. And then we just wait for the next speaker. Thank you very much