Open Meteorological and Climate Data

Video thumbnail (Frame 0) Video thumbnail (Frame 658) Video thumbnail (Frame 1187) Video thumbnail (Frame 1722) Video thumbnail (Frame 2249) Video thumbnail (Frame 2746) Video thumbnail (Frame 3214) Video thumbnail (Frame 3634) Video thumbnail (Frame 4166) Video thumbnail (Frame 5022) Video thumbnail (Frame 5479) Video thumbnail (Frame 5967) Video thumbnail (Frame 6406) Video thumbnail (Frame 7141) Video thumbnail (Frame 9265) Video thumbnail (Frame 9832) Video thumbnail (Frame 10418) Video thumbnail (Frame 10895) Video thumbnail (Frame 11345) Video thumbnail (Frame 12052) Video thumbnail (Frame 13134) Video thumbnail (Frame 13688) Video thumbnail (Frame 14599) Video thumbnail (Frame 15031) Video thumbnail (Frame 16243) Video thumbnail (Frame 17026) Video thumbnail (Frame 17470) Video thumbnail (Frame 18034) Video thumbnail (Frame 18598) Video thumbnail (Frame 19394) Video thumbnail (Frame 20050) Video thumbnail (Frame 20890) Video thumbnail (Frame 22036) Video thumbnail (Frame 22681) Video thumbnail (Frame 23639) Video thumbnail (Frame 24260) Video thumbnail (Frame 25188) Video thumbnail (Frame 26027) Video thumbnail (Frame 26888) Video thumbnail (Frame 27435) Video thumbnail (Frame 28606) Video thumbnail (Frame 29365) Video thumbnail (Frame 29838) Video thumbnail (Frame 30548) Video thumbnail (Frame 31295)
Video in TIB AV-Portal: Open Meteorological and Climate Data

Formal Metadata

Title
Open Meteorological and Climate Data
Subtitle
Building Bridges between user communities!
Alternative Title
Free and Open Meteorological and Climate data - what is missing?
Title of Series
Author
Contributors
License
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
2019
Language
English

Content Metadata

Subject Area
Abstract
Copernicus offers, besides the well-known Sentinel satellite data, a wealth of domain-specific open environmental data sets, e.g. data on climate, wildfires, air quality, floods. One of the most popular data set useful for many environmental applications is the climate reanalysis ERA5 produced from the European Centre for Medium Range Weather Forecasts (ECMWF). Improvements in the spatial and temporal resolutions lead to an increase of the entire data volume up to 5 PBs. Additionally to the sheer amount of data, meteorological and climate data have a certain complexity, especially for “non-expert” users, as data can have up to five dimensions and two time dimensions. The current situation shows that a full, free and open data policy is one important prerequisite, but the key to fully unleash the potential is making the data ‘accessible’. If open data is not accessible, it becomes open data that is locked away in large data silos. However, making meteorological and climate data “accessible” means more than just improving data access. It requires improvements and developments along the entire data processing chain, including the development of example workflows and reproducible training materials as well as developing / enhancing mainstream open-source software tools. In this context, the FOSS4G spirit is vital. This talk puts the spotlight on open meteorological and climate data. Current ‘accessibility’ challenges and future needs will be discussed in order to make open meteorological and climate data better accessible to everyone.
Keywords Keynote

Related Material

Video is cited by the following resource
Freeware Range (statistics) Parameter (computer programming)
Goodness of fit Logic Universe (mathematics) Moment (mathematics) Open set
Voting Building Open set Open set
Service (economics) Integrated development environment Logic Prediction Range (statistics) Open set
Area Beta function Angular resolution
Subject indexing Angular resolution State of matter Parameter (computer programming) Parameter (computer programming) Extension (kinesiology)
Subject indexing Goodness of fit Frequency Information Angular resolution Image resolution Temporal logic Building Moisture Code Price index Moisture
Channel capacity Image resolution Angular resolution Temporal logic Mathematical analysis
Freeware Open set
Service (economics) Freeware File format Virtual machine Usability Data structure XML Open set Data structure Virtual machine
File format Kolmogorov complexity File format Usability XML Kettenkomplex Open set Virtual machine Latent heat Mixed reality Data structure Data structure Physical system
Satellite Covering space Standard deviation Service (economics) Observational study Kolmogorov complexity Projective plane File format Civil engineering Database Volume (thermodynamics) Open set Vector potential Formal language Medical imaging Arithmetic mean Term (mathematics) Single-precision floating-point format Object (grammar) Laptop
Volume Logic Term (mathematics) Information retrieval Data storage device File archiver Mass Term (mathematics) Physical system
Arithmetic mean Group action Service (economics) Information Multiplication sign Projective plane File archiver Video game Term (mathematics)
Point (geometry) Pixel Focus (optics) Octahedron Divisor Musical ensemble Field (computer science)
Point (geometry) Latent heat Standard deviation File format Term (mathematics) Real number Point (geometry) Field (computer science) Pixel Traffic reporting
Standard deviation Group action Service (economics) Software Logic File format Multiplication sign Self-organization Term (mathematics)
Software Core dump Expert system Wave packet
Onlinecommunity Dependent and independent variables Natural number State of matter Term (mathematics) Operator (mathematics) Decision theory Workstation <Musikinstrument> Self-organization Traffic reporting Number Twitter
Greatest element Latent heat Matching (graph theory) Process (computing) Information Software Decision theory Multiplication sign Energy level Expert system Kettenkomplex
Point (geometry) Pi Word Decision theory Multiplication sign Software testing Expert system Physical system
Complex (psychology) Latent heat Logic Kolmogorov complexity Energy level Dimensional analysis Pressure
Complex (psychology) Web portal Web portal Channel capacity Multiplication sign Archaeological field survey Channel capacity Moment (mathematics) Fitness function Limit (category theory) Process (computing) Process (computing) Endliche Modelltheorie Volume Hydraulic jump Annihilator (ring theory) Computing platform
Confidence interval Maxima and minima Volume (thermodynamics) Open set
Shift operator
Type theory Word Group action Service (economics) Googol Zeno of Elea Different (Kate Ryan album) Code Range (statistics) Text editor Open set File Transfer Protocol
Onlinecommunity State observer Discrete element method Plotter View (database) Moment (mathematics) Cartesian coordinate system Google Earth Programmer (hardware) Googol Different (Kate Ryan album) Prediction Endliche Modelltheorie Physical system
Metre Group action Weight Library catalog Parameter (computer programming) Cloud computing Subset Googol Process (computing) Operator (mathematics) Order (biology) System programming File archiver Data conversion Physical system
Standard deviation Service (economics) Arm Mapping Moment (mathematics) Parameter (computer programming) Formal language Volume Sign (mathematics) Process (computing) System programming Data conversion Communications protocol Physical system
Latent heat Building Focus (optics) Software Device driver Client (computing) Device driver
Email Dataflow Electric generator Device driver
Mathematics Programming paradigm Service (economics) Service (economics) Internet service provider Projective plane Moment (mathematics) Archaeological field survey Point cloud Cloud computing Perspective (visual) Point cloud
Mathematics Programming paradigm Service (economics) Message passing Mathematics Programming paradigm Multiplication sign Point cloud Energy level Cloud computing Physical system Point cloud
Message passing Shared memory
Electric generator Code Multiplication sign Shared memory Code Mereology Wave packet Wave packet Bridging (networking) Operator (mathematics) Universe (mathematics) Authorization Thermal conductivity
Building
all the army he introduced the first keynote speaker this morning you really have argument comes from the european centre for a medium range weather forecasts she's been working there are few years currently doing a ph d. on the same topics yulia welcome you have the floor.
like a good morning everyone i'm quite surprised to see so many after seven hours of open bar so it really gets. so yeah my name is too early i am a visiting scientist at the moment at least m w f one else was doing a ph d. at market university and for the next twenty minutes i would like to the put your attention to metro logic and climate data because they are openly available and they ready for you to use and i also would like to.
use the opportunity to talk or to to talk about some aspects of vote open data but before i started to actually knows who bought the zimbabwean is still european center for me to manchester for costs please raise your hands three hired to seep into ok ok so a few.
ok that's that's already good soul and his am doubly f. is primarily a numerical better prediction center we provide better for costs to national metro logical services.
and but we also are operate to the company because services to a panic as climate change service and the company has at most few monitoring services and because of these two services we have also now a full range of open environmental data available we have for example at data on climate available.
the most popular betas that is to your of fife realises which just has been published in the beginning of this year it is our early data on a thirty kilometers spatial grit and it's going back so far under nineteen seventy nine but it will soon go back even until nineteen fifty and we also have seasonal flu.
because stay top monthly for cars data up to six months the hat but not only have climate data we also have a quality data and this is quite recent because of the violet fires in in in the amazon the area.
so does as biomass of fuel index which we see here and so it's really helpful to monitor the impact what's actually happening here and we have different parameters an air quality on also and on carbon dioxide and nitrogen dioxide and we have three analysts the state.
so they go back on to two thousand and three but we also have for ca status three hours before cost up to five days in advance but it's not everything we also have to data on fire danger we provide a serious of them fi of eta in does a fire danger in disease for example fire but the index or find fuel moisture co.
what and they help you better to assess the possibility of the afi occurrence them somewhere and last but not least we also have very good data on floggings so and specifically and river discharge information and we have forecast a tough for their daily on attend.
kilometers base the grit and we also have a real analysis going back until nineteen eighty one and so we have to use and the good news is that these data are all full free and open available under capacity because that's exciting isn't it.
so i'm but in preparation of the talk i was thinking about like i'm some people already asked me ok but does it actually mean full three and open and i looked up some like a deficient open daytime and i have fallen a definition from the european data poured low and.
so what i have found very interesting his stead day i'm sept at aspects like formatted structure and machine readable a t. make today taught more usable but they also said that it does these do not make the data more open and i want to actually turn around just quit.
question today. and ask two aspects of non inter abilities so that data are not easily into exchangeable between system or data complexities so we have got the structure of the data is complex and we have complex met a day died start to understand and community specific format so.
do at these aspects make the data left open.
so when i started over and i have joined easy m w f i was involved in the project were we so we knew that the open data our we its growing in volume and it's getting harder and harder to actually download the data and so i was involved in a project where we investigate it and how we.
can actually provide them a better on demand exes to the day top based on data standards so we implemented a vet coverage service for climate and metro look single data and this look at this is how my day to day every book liked because i tried to fit in metal object going climate data. into a standards and also into a technology which was primarily developed for earth observation data and satellite images but it worked so we have generally it and we also see the potential for that covered services and standards. so but it also showed me that like working with different partners and the project is despite the fact that riso probably talk about me to be speak the same language and we also use the same terms it often also doesn't mean did we actually have to say meaning for these terms and so i would like to share with you.
today five facts about the metro logical in climate community to better understand. mass so the first fact is picked data is not a new term so if we define big data just by the sheer amount of data used in w f is quite experienced and handling storing and archiving large volumes of data we have to metro logic and archival retrieval system it's a cold mars stark.
wife and kids having its own in them are psychiatry has more than two hundred fifty petabyte of data. as stuart and it's as the eye chart largest archive of metallurgical data old fight the second fact is operational means really operational their love of services and projects out there de claim to be a to have the operational service by offering data.
org within twenty four hours only one group days for the same top yes and a mentor mitrovica community operational means it has to be up and running twenty four hours and seven days a week because for cost they can save life so it's very vital to disseminate used data and information and time.
such factors we talk about fields and grip points not bands and pixels so i'm add the focus d'etat produced on oct a huge will grit and so far cost data are basically just ballot for this one specific gripped point and if you retrieve data on a regular let you.
did you don't get to grips it's important to note that the data between two great points to a day have been into plated and the real forecast value is just valid foot is specific report and discipline act is standards and interoperability are no new terms so we as a common data format the the medical community.
he liked its the group of four met and does is like a very efficient data format to into exchanged a top notch logical and four cars stayed up between metro logical organisations and so it's its it's very efficient data format but it can mean dead people who are not familiar with reform it.
it mind might meet need it might mean did you have to invest some time to actually better understand and how to handle it. but are for expert uses and four national medical services it's not a big deal because just brings us to affect fife we like custom built software developed and house because does self-service to tailor the software specifically to our needs and also to make the handling of group died.
data very efficient and four is an obvious core users and expert users like train meteorologists.
the employees from national natural have to go to the organisation's this is a it's not a big deal and its it's a very good because many bergen linux book stations but we also day we see a trend now with a panic was derelict larcher user community day actually interested in the data. and this just this can lead to lead to a problem and so. the decision also what the european commission and found out they published a report the company has market reported the beginning of this year and a state that number of one trend is step to daisy a diversification of users and and to months but the definition of users is this quiet operator.
i would say probably if i ask ten people here like was a data user i probably get ten different responses they are different terms i would be or what users can be so deccan be an end user can be decision makers the intermediate users but the problem with the.
some users if we try to actually match them on what level of the two spatial processing chain date probably belong to us so it's good to permit is going from raw data on the bottom up to the prostate time and then generating more information it's very. the difficult actually match and also probably depends on the level of experience how you what to put specific users if someone has a lot of experience using she was spatial daytime and developing software if someone who just developed workflows.
and pies and maybe you would put him or her as intermediate user but someone who the test heard about use facial data and uses the reactor in it yes systems and and then someone compared to someone who already he develops workflows and tyson.
he or she is already a much more advanced so the point i want to make is like it's very vital to actually understand a word or users because then systems and also did the data can be better opened up but his same time as also like a very challenging so there is some to challenge.
this excess so it's really go back to metro logic in climate data new data users stay might face and us some challenges and one have to. the most important challenge i would say is data complexity so we all knew about a three dimensional data but much luck people data we can't even go up to four dimensions and and even five dimensions if we talk about osama data we don't trust one for costed one specific.
i am we actually lead to run the model fifty one times to bend generate to demean out of fits and this is really does have a much more reliable and four cost so data complexity of his is a challenge for new data users i conducted a user requirements of your the beginning of this year.
and because i'm interested in who are the user's what tools day i use at the moment but also what challenges stay faith and the fourth biggest challenge is to identify it is limited processing capacity to growing data of all human data are discriminated and the non-standard. diced way and that they are too many platforms and portals did users are just confused they don't know where to find a taco to retrieve the data.
it's probably growing data volume it's not the big for the big surprise prizes but these challenges are very important to overcome and also to realize because if we don't specifically addressed and in which just continuously. aggregate in more and more data and we have open data available in large data as i lost but it's actually open locked data up because no one can find them no one can access them and no one can use them so they're the debt has been like kind of the competition i feel i'm confident.
francis dead yet people it is a i have a generator d.'s indies a terabytes of data every day and all know we generate even more terabytes of data i have every day and i would like to turn it around or encourage everyone rotted and to think of the eight we generated as so many is so much state.
so out to also asked ok how much of his open data that is actually produced is used and just to make the data useable one specific prerequisite is to provide it to do a gift data access to it and there is a very important shift we have to go to from pure down.
notes of business to more on demand data services and so we have a range of the different types of data axis a waste of them for also the data i introduce the beginning and that the most are still download services but then now with a panic as and also have to set up. of the climate data stored there is a is a is the past two words more on demand data services for example did climate data stored toolbox bill is similar to google earth and shouldn't have a online code editor you can directly access to data you can generate your group's low.
and pious and and then you can build about application or a was to lose asian and you can just as a few a plot so one week but it's also important to the single off debt yet we have different views of communities and different user communities use offer.
in different systems so for example of google earth and she has used a lot of to earth observation community there's an example from doubled food programme they develop at the moment a flap prediction model on a on google earth and shun but they also interested actually in using your of fife data. and two to make the model better but the problem is look at how to do how do you actually bring it together you can try to insist the day to have its if the does this dems are not into a purple it can be challenging.
at so and that's why we and decided to order. i also believe in debt like as long as the data systems are not into operable dead mere archives of data are can be also britches between different use the communities and so i've been working on making at a small subset of seven era five per meter is available on google earth and ten.
the process to make them available is a very good example how systems are not into a purple because in order to make data from one system climate data stored to the goobers incheon to make them available dear we have to download the data in group or nets e.d.f. we have convert data dated to do your turf we have to. oh damn to google cloud platform and then we have to interest them to google earth and shun and so the entire process to make seven year a five per meter is available around for a five terabytes of data it took around nine months and this brings me to do meets and on important need is yes.
systems have to in to operate with each other and data have to be easily exchangeable with each other and this can be achieved with standards and but it has to be beyond only has to go beyond that mapping services because we are data users and large volumes of data we want to have to real.
taxes so i'm we also have to we need more but covered services and the processing services to second need his arm yet we have to make it easier to handle and process the data with tools users use and so placement are at the moment the the language as a data signed.
just use and so yeah we need to a good and have handy to represent drivers to to book with these data and isn't that left has been a to put a strong focus on making it easier and also bringing to custom build software to the pious in bolton a specific.
sadly the last year and a third need as we have to use tools and the packages and the data we also have to show how we can actually efficiently put to use everything together so we have to generate reproducible book flows and you have to train the data if you like to discuss and.
talk about reproducibility a piece adam have a look at the reproducibility bookshop it easy am doubly after we will dedicate three days so on this topic in middle of october.
and sell now it's a question is ok we are we going and it's not a surprise yet cloud just a future so is improperly s. is involved in quite a few cloud project at the moment so but this shows shows also that there's still a lot of question marks as well how data.
services can be set up also under perspective from data providers and but the good thing is to use a survey i also ask users if they would be much. she made it to migrate to cloud services and sixty eight percent were interested or even very interested to migrate to cloud services to do to processing dear.
but we have to keep in mind not just the just because of the fact we have a new system we have cloud services and say yeah and it might be beneficial for users it doesn't meet necessarily mean that users also used to the systems and so cloud is a paradigm change and we have. to keep in mind to change also always takes time so to conclude this talk i just would like to give you three take home message just and the first one is a quote from about einstein saying problems cannot be solved by the same level of thinking that created them so i just wanted to say that we have to.
keep in mind in our day to day of her rod and to report on what is possible now to think where do we want what do we want a half in ten and twenty years and work towards does pot to second take home message is i think it's less a problem or less of a problem you just community to make reproducibility and sharing.
to a pirated to a part of your personal code of conduct those see it as a time rather than a hassle and a third on his train operators share and to share your skills and also knowledge so what we discussed doing these days i'm here and also not a conference is we have to bring to universities a we have to.
going to companies and the public authorities to really train the next generation of to use facial practitioner us and i like the the to the the resemblance from vastly yesterday so yeah and i want to concluded with the at just continue building bridges and i wish it was everyone and inspiring.
was which he thank you.
i have.
Feedback