Exploring large amounts of weather forecast data through Open Source software

Video thumbnail (Frame 0) Video thumbnail (Frame 1342) Video thumbnail (Frame 3766) Video thumbnail (Frame 5092) Video thumbnail (Frame 5730) Video thumbnail (Frame 9104) Video thumbnail (Frame 11166) Video thumbnail (Frame 15571) Video thumbnail (Frame 16872) Video thumbnail (Frame 18361) Video thumbnail (Frame 20397) Video thumbnail (Frame 23604) Video thumbnail (Frame 24998) Video thumbnail (Frame 27205) Video thumbnail (Frame 28037) Video thumbnail (Frame 28928) Video thumbnail (Frame 30881) Video thumbnail (Frame 33503) Video thumbnail (Frame 34645) Video thumbnail (Frame 35855) Video thumbnail (Frame 36803) Video thumbnail (Frame 42890)
Video in TIB AV-Portal: Exploring large amounts of weather forecast data through Open Source software

Formal Metadata

Title
Exploring large amounts of weather forecast data through Open Source software
Title of Series
Author
Contributors
License
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
2019
Language
English

Content Metadata

Subject Area
Abstract
ECMWF is an operational centre, producing global numerical weather predictions. ECMWF also operates many services for the EU Copernicus programme . ecCharts is a suite of web based services to visualise and explore ECMWF real-time forecast. The service offers a Web Map Service (WMS) as well as an interactive client application which provides forecast data for meteorological community including Copernicus services. Recent development work involved the replacement of the client application with OpenLayers and implementing a caching system with MapProxy. This posed many challenges due to size and frequent updates of the data, combined with the challenges of offering a fast and smooth user experience. The work also resulted in contributions to open source projects, like OpenLayers, which will benefit the wider user community. This talk presents the benefits of moving to open source community software,the architecture of the updated system and on the challenges and the solutions that are taken when dealing with big and dynamic data sets both at the frontend and at the backend services.
Keywords General

Related Material

Video is cited by the following resource
Beat (acoustics) Presentation of a group Open source Visual system Multiplication sign Range (statistics)
Computer program Open source Multiplication sign Mereology Neuroinformatik Product (business) Local Group Prediction Web service Operator (mathematics) Core dump Service-oriented architecture Endliche Modelltheorie Physical system Area Focus (optics) Software developer Moment (mathematics) State of matter Computer program Internet service provider Heat transfer Range (statistics) Cartesian coordinate system Supercomputer Mathematics Data management Process (computing) Software Visualization (computer graphics) Web service Network topology Internet service provider File archiver Service-oriented architecture
Point (geometry) Metre Game controller Open source Image resolution Multiplication sign Time series Menu (computing) Event horizon Product (business) Revision control Data model Optical disc drive Roundness (object) Term (mathematics) Operator (mathematics) Visualization (computer graphics) Software Energy level Information Endliche Modelltheorie Pairwise comparison Area Focus (optics) Touchscreen Information Mapping Interface (computing) Open source Parameter (computer programming) Cartesian coordinate system Product (business) Degree (graph theory) Process (computing) Visualization (computer graphics) Clique problem Service-oriented architecture
Greatest element Scripting language Multiplication sign View (database) Plotter Numbering scheme Set (mathematics) Insertion loss Disk read-and-write head Dimensional analysis Neuroinformatik Web service Roundness (object) Envelope (mathematics) Different (Kate Ryan album) Hausdorff dimension Vertex (graph theory) Endliche Modelltheorie Physical system Scripting language Mapping Data storage device Price index Storage area network Front and back ends Product (business) Degree (graph theory) Process (computing) Series (mathematics) Website Self-organization Reading (process) Open source Time series Library catalog Axonometric projection Field (computer science) Number Cache (computing) Causality Software Service-oriented architecture User interface Operations research Axiom of choice Information Tesselation Interface (computing) Projective plane Java applet Database Word Visualization (computer graphics) Logic Connectivity (graph theory) Musical ensemble Service-oriented architecture Buffer overflow
Constraint (mathematics) Divisor Constraint (mathematics) Multiplication sign Image resolution Client (computing) Field (computer science) Revision control Arithmetic mean Process (computing) Software Revision control Graph (mathematics) Speech synthesis output Divisor Resultant
Multiplication sign Image resolution Computer-generated imagery Similarity (geometry) Set (mathematics) Insertion loss Rule of inference Neuroinformatik Medical imaging Term (mathematics) Single-precision floating-point format Square number Hausdorff dimension Energy level Electronic visual display Focus (optics) Information Mapping Tesselation Interface (computing) Moment (mathematics) Graph (mathematics) Bit Similarity (geometry) Mathematics Category of being Frequency Information retrieval Statement (computer science) output Family Buffer overflow Resultant
Open source Observational study View (database) Tesselation Insertion loss Client (computing) Mereology Axonometric projection Usability Front and back ends Data management Medical imaging Mathematics Cache (computing) Operator (mathematics) Software Vector space Touch typing Utility software Smoothing Operations research Electric generator Mapping Tesselation Server (computing) Interface (computing) Physical law Projective plane Gradient Open source Client (computing) Control flow Software maintenance Open set Arithmetic mean Data management Raster graphics Personal digital assistant Synchronization
Tesselation Multiplication sign Client (computing) Term (mathematics) Medical imaging Mathematics Arithmetic mean Process (computing) Cache (computing) Synchronization Personal digital assistant Different (Kate Ryan album) Function (mathematics) Synchronization Musical ensemble Hausdorff dimension Information Endliche Modelltheorie
Point (geometry) Implementation State of matter Multiplication sign Plot (narrative) Dimensional analysis Neuroinformatik Revision control Mixture model Latent heat Mathematics Web service Bit rate Different (Kate Ryan album) Operator (mathematics) Artistic rendering Hausdorff dimension Electronic visual display Energy level Metropolitan area network Physical system Mapping Information Tesselation Weight Parameter (computer programming) Client (computing) Bit Flow separation Front and back ends Demoscene Process (computing) Personal digital assistant Synchronization Arithmetic progression Physical system Thomas Bayes Row (database)
Operations research Beta function Feedback Multiplication sign View (database) Feedback Code Client (computing) Bit Client (computing) Cartesian coordinate system Sign (mathematics) Category of being Web service Process (computing) Human migration Cache (computing) Velocity Revision control Website Process (computing) Implementation Proxy server
Presentation of a group Implementation Open source Software developer State of matter Real number Direction (geometry) Insertion loss Open set Information privacy Mereology Rule of inference Dressing (medical) Computer icon 2 (number) Web service Energy level Integrated development environment Physical system Area Focus (optics) Mapping Interface (computing) Projective plane Keyboard shortcut Moment (mathematics) Shared memory Planning Stack (abstract data type) Cartesian coordinate system Message passing Uniform resource locator Process (computing) Web-Designer Speech synthesis Website Service-oriented architecture Arithmetic progression
the on a good start our and was broken only.
in mr ritchie and saying is and who are exploring large amounts of it there before because data to help us or so for making things. a little the you can hear me in the hi my name is chair on a iraq at the same the body of the minute presentation for the next twenty minutes is about exploring large amounts of data for cost a time through open source so for so i lay introduce the same probably have a beat and then i will. to what we've done the to modernize our the main base to interactive visualization system called easy child's and of what the steps we've taken the tray cheated so who are easy and oblivious to the european centre for medium range with a fork us a cell research institute to.
the developer the weather forecast models but is also a twenty four seven operational service provider so the the data to regenerate out of our motto is four times a day it has been a day he is given out to our member states to co-operating stays and a broader community a. full so that they can use are all data to integrate into their downstream processing and then he should the focus you see on t v. or your telephones for example we also prayed one of the largest spare computer facility a because we needed to be able to do with a model for the forecasts are put on a timely manner and you also operate the one of the largest metallurgical date archive in the world. at the moment the keeper on tree hundred petabytes of data here in archival system the area are based on reading for the time being in united kingdom and this is our core facility but you also prayed to services from the used a panic as program and one of them being the monitoring service have taught most fair. ed their school to calm service and also panic as climate change air service which is c three s. he also contribute to a panic as emergency management service save which is a win by a year later flight and his program. it's called the farce.
i am part of the web services team a it easy m.w. of what we do is that ok be generate is the data and the push to our customers but he also has some applications to replace the implications and services that are and customers can access to data immediately to develop it. his stand out that interfaces so we have a processing complication is asian system the an interactive system on the weapon that i will the talk in detail later every have some very baby i should download the data by using standard a.p.i. tools we have lost so flaws of graphical products that region. married every day and an operational men a and b. provide other weapon a services the based on the ward easy his services like to be a mess. so distill these mainly about to their base visualization application the april i to our customers and this is called easy charts and are enforced to modernise it today by using open source software.
so these are some the screen shots from our into each as interfaces you see taste very colourful pretty pictures are so what exactly is easy child's the seven days application to inspect and relies on the same that we have data and plastic a panic as services.
the from the most very monitoring data so the also provide all are they is and data through to the dublin mess the graphical the products he's an immediate access to the to the maps which is very important individual issue the community especially in the event of orcas it has interactive feature. is that users can zoom pond clique extricating information as a in the interface it has use of control visualization because users can generate their own products by combining lay is out of two hundred fifty layers we offer and they can design a product and and say this product full will use it. the to use it later basically every hair customizable were permit is as well use m.w. of france operational model that he called a high resolution model but you also generate it probably stick focus so the round as a model fifty times every day twice a day and odd of those fifty four. costs which is slightly lower resolution to users can generate probably stick focus so they can generate for example the to me to temperature probability. two metre temperature being more than ten degrees or less than your degrees they can do all those custom generated products out of five or so the interface can do to the global maps so the only work on the global level soul our data are global and taken the agency any. the forecast eight old around the world and they can also do the point based post process data in terms of to time series or they can click anywhere on the world and they can see time series a lot of two of those probably stick for prostate or so he had to stand out on the c.w. meserve is every liberal as well. and he also offered to our customers that are using our interface it to be able to compare different versions of the model which is a very useful for them too. so he tries to do a lot so this is a screen shot of the application we have and maximise the area have to have to have today to delay is the users can access to data own products to the menu.
we have fair different projections a pull our store graphic projection is very popular in a metal visual field so this kind of compulsory to to have it in any metallurgical interface and every had loss of data inspection tools ready users can click can read today tile extract the data of these point. evie have a interface tool for delay is as i mentioned the head like two hundred fifty different layers the and then they can combine all those day is a polite different styles different colour schemes they can apply those a complications for example in the publicity for cars so they can generate all those are the cause. system. the maps as they like it and very important in the metro logical communities that time dimension we have a navigation of the time to mention the at the bottom of the interface where users can navigate to any violet time have to have to four costs by using these in. looting animation as well. so just before going a into detail what we've done the true the modernization of are so for a is to give you some information about our infrastructure v.v. run in house a cluster of the computers to the to host all those services we have.
the data still that you call the data a year and to be still green fields because our model generates automatically agree feels degree is the for martin demetri wish to world which is governed by to work metallurgical organization. the we have a special databases for time series data so we basically take all the city and metallurgical feels and and transform them into the into time dimensions of the taken be generated a very quickly he keep all the metal data in mongolia. he led them we had a computer notes the beach we do all the work number crunching business is happening that it's what we call to service a year so the loss to the envelope micro services mostly written in pipe him and managed by distillery workers editing lose too rich you did. a time compute whatever is needed from the uses requirements and them the u.c. san diego is on the processing tool which is called easy courts the to do that. and then be at the end of the overflow regenerate plots and it can be used these him that lives magic sofa to generate the plots so they can to on the set is high it is here is jungle. so jungle rounds those were close the that is whatever is needed to become the on the on the service they have a views rabbit m q the to orchestrate old word the and views medication before the casing a on to serve the site. so all this information the met had a tad it is needed to be able to generate any lay is this a good temperature the. we keep them is jason pile see a in a database and music and more goody be to do that the day which is very very nice and the user interface is a indication is jealous scraped the main a cakewalk he is so if you look at our infrastructure actually use the on the service. site all open source or so fed which we are very happy but our main problem was there on the user interface because he's all basic a callous script all the g.i.'s kind of information we've done it at a house the at the time he developed it is a need to pose as many. the shoes.
and that brings me to the the. justification for the modernisation of to have to call the first version of the speeches has been written about ten years ago so although zooming planning and j.s. related called we've done it ourselves and. and in this ten years' time our data for a given field there is a lucian of the data grieve in the indy order of to the is a factor of ten. and because our data is quite big actually and and if a quincy to update the data is very often this twice a day he we had to do most of the things on demand and with the result is an increase in the in the data of course everything is lower compared to ten years ago. so growing data sized plus the time constraints we have we have only one hour to prepare to data even less than one hour to prepare whatever is needed is it like to see being or caijing of the data.
and d.c. is not enough to prepare them so he had to do all the processing on demand to know who also like the problem is probably stick for cars that needs to be done. from the input from the uses so he's completely cosmides build meaning that the country like a ship will prepare anything even if he had to time. so those two things to speed because of the growing resolution of our data and also to maintain ability because we have all these g.i.'s related to a quarter return bara by ourselves. there are the main two reasons that we had to the modernisers of it.
a bit more information are metallurgical data so i mentioned earlier that we have around two hundred fifty lanes in each layer has a time to mention a similar to this one so on the on the rose you see dates those are the forecast based time so the vendor focus is made to pay. basically and as close to see the focus when the time so when the focus is while it so they say each of those a little squares is a representing a data set for a given lay a so we have two hundred fifty of those similar pictures in terms of two data sets. and even more challenging is that. each of those rules are updated twice a day so the everyday the we have two new eve roles of those two hundred fifty layers arriving so he stayed the challenge we face in the metal is cool forecasting community is slightly different than the g.i.'s community is because our data. it is not may be as high resolution is the u.k.'s some the giants statement but is a updated very a quickly so it is twice a day so it gives us very very little time to do anything at once. a.
and of course are the first approach ten years ago to make sure that you always show the eleven data was that you always generated the maps as single tiles so for example here here's an example of the overflow what the two to generate a map is. that the received l.s.a. requests six hundred to sixteen hundred by nine hundred to excel temperature property and precipitation and this is what we need to do in our big can be had repeated data for those two different way as he had to do the computation a store. both the based on the input from the user and because they might be asking whatever the the proper today they would like to see an immediate are plotting so for each way of the to generate a map and then be random togethers of the basic is my passion together to generated final png. families are this one so day ten years ago that was working fine when the data sizes were manageable but it's very slow at the moment because you have to because of to the result is an increase and also the interface see if you change a midsize those whom level little boy. for bonding bucks tended to the sighting partake in the other positive done on the global level so retrieval of the data all compensation is a polite to to the data for the whole globe so they are caged in the first request but the pluses to be done again and again kids we. we don't have tiling concept. and also the are always as fast as the slows later the display so if you have an expensive a year it requires loss of data the to be retrieved and and computed and it will be relatively slower than the other way as and and because we had to wait for the rendering to generate a final image that will be. he is the speed that are interface was able to the and produce.
so to attack to those two issues speed and maintain utility air we knew that he had to employ the tiling but it poses laws of the shoes be to metro original data the that i will to be later. so we wanted to be implemented tile in so that we can use the occasion much more efficiently there and also the smaller in size as means that the each image each child will be generated much faster. it also gives much smaller to finance them operations as well so looking at the interface of easy just be all on he wanted to change the demand generation part of it all the other parts of the use interface our customers for happy so he didn't touch it solo effort was to grip.
these are the existing it may pay handling with the open source of the soviet the some studies about a year ago he decided to use openly is to do that. and then you start from the all players from the client side and make a living changes on the back end because the he had to to loss of changes had taken to accommodate the tiling as well. so your place because some clients of the door firmly as or players in our case is great ateliers many of the features that weaver after the price and tile management which is essentially to our case for a week can be extensible so we can all right of hunches and a change of behavior is free. the as our requirements this i just a week is a very easy handling of to projections which is again essential for us. it handles different sources a map sources that william s. these are the primary source views are on the p.m.s. servant to serve our day to the beach which was fine then we have some director tiles as well and some raasay images as well so often the iss grade it can handle all those things. the the usability is very nice it and there's a great today community support as well. he said.
so all those changes on the klein side to add. pausing some issues for us is the main issue was assigned grooms asian off the tiles because if you are serving as a single images may be you don't need to do much about the tiles but when you are generating the tiles on to mont and they all come on different times because they all need to. front and model processing needed then the second is in synchronisation of tiles are essential in our case otherwise you can show a mixture of different to tiles from different time steps which is very wrong in our case so we put a little for to synchronize the tiles.
the and also the do a lot of animations i use as a like animations and they like to step over the time to mention meaning that we need to also synchronize the tiles the third time israel is a win as well as delay as well. so there was the main it.
and forget the had to put to be able to make the tiling business rate for us especially full tiles that are generated on demand. so the bank and improvements because of the say tiling approach the was slightly different he had to change some of the dublin miss specifications to get our needs a kind of extending a especially handling to time dimension and he had to do some kind of preceeding of the tiles so. but we can give our uses a petty use experience and to do that what we did essentially is that whenever a use a displays a child which is a mixture of several layers in our case a beach each of ladies containing many tiles in the bay can't be get this information what. the day at the scene and then be generating next lifetime steps off those layers for all those titles which are very nice for us because by doing that is because i used to spend a bit of time looking at the information on the map. and by the time they would like to move on the time dimension to make staple to step off to those ties already ready and dedication so that the there was a nice approach to from our point to the it to pick and.
devi had another problem we take aging system that we are using which is man cation he is a very nice to very fast and efficient. but because of the way the change we hear instead of generating single tiles not regenerate many tiles that these tiles need to be synchronized again on the service side as well. because we do all those retrieving data and computation operations on a global level so when you have fifteen of the same a delay their fifteen different tiles arriving there is actually asking to the pecan to do the same job fifteen times have you realize very soon. a very quickly that we are crushing the system by just simply asking if you time steps so we had to implement tech kind of looking system of m.b.a.. then the get although style record is the first one generates a local while doing the job in this case for example retrieving the data and other tile stages weight and keeps asking to the medication he is the state already will not pay and them as soon as it's rated a progress is the editor. miles and and the computation is done by the same tile floor but some of the child doesn't really matter which is creating a look at you can afford to step and and the others the weight on telly the computation is on his well and then each year tiles to the own plotting be. is this the only difference a compared to each child for a given day. and so to sum up the to be ended up no is a better version of the e.c. chas the beach we are very happy because he's feeling much pasta and smooth compared to previous version a.v. have very positive experience for can be open areas.
it was very nice experience for us the so the managed to make the application process but we also got rid of our own g.i.'s related court he said the e.u. now use openly is for example in decline site which is great so he can do updates a much weaker than they used to be used to. two and now we are getting some feedback from our users better uses the the pacific take as well as but the work is not finished yet we still need to migrate to have only a six.
that will happen after the end of this year i believe every have other applications as well so we can apply the same approach to those other applications to a v's should start preceding of the population is we have on the very limited time but we can still do a bit of preceding it. at the bank and every would like to implement the occasion on the service site by using them proxy and them view also experimented some my client side processing like those properties i was mentioning earlier but he can't do that on the client site rather than going back to deserve it. so this it the.
we also have an opening it to support our ethos the flight and the system the project we are looking for full stop web developer to earth does a is not closing on the second of september the spin extended now to sixteen to of september's so if you're really fun a few is interested p.c. and. play. as you can use the same the benefits i to find the it job opening of a thank you for. thank you very much an interesting go presentation we still have five for wanted to question before you move to under talk so if you have any questions place.
oh that looks very nice a on his than the plans to open sourced is all three went. so sorry. we have in the pants to open source to call dirty have been developing. the icon to return to open source to cold like to share the cold and nine. his am doubly of shares most of his court including the e.c.a. easy course speeches process in the crypt data show a magic speech is doing to plotting the speeches is kind of binding into the infrastructure so ever some polls in the past but i don't know if the moment what will be but we have no problem to share it. is this how the same that we have normally risks. thing. was it. thank you very interesting presentation and you've been talking about the old easy with services and in that world you are using to your interface but to do you share or do you plan to make them accessible to. the other parties has two other occasions the well you we have a example it has some public data the at the moment and he's already available in w m s that could be used by other applications to come stated the utmost very monitoring service state or from the. the message is also available for any public use and the full at the data the real focus tater you need to have the access to data in the first place so you need to register and go through the process so that you had access to our data. then you can studying and you can start using t w ms service as part of your application for example. but there's also a loss of talk at the moment about open data so this is the direction he seemed as if he's going as well as soon as there will be progress on the site eventually i guess you seem to be of data including all the services will be open at some stage in the future. the higher you mention the use of our maps service i'm just wondering the are using any a tools for like open source tools are the builder the b.m.'s yourself in the build a double him as ourselves this again almost ten years old so i think the same the year was one of. the pioneers of to the problem is implementation in a mental issue ruled. you just build it ourselves we don't use a new tools that i'm here. sayers locations and the thanks most the player the players available in the sea charts are those a global coverage or does that limited to a year at everything is global so this part of our issues as well because we can do processing much fossil on the global level and once they have done. they're available for all the other uses but we need to synchronize those processing said to pick and so everything is globe live a little all the fields. thank you very much you can still have a dress him has some question. you still being if you want or drop by their booth was well yeah we had erred by the way we have bluetooth the in the main areas explained was the fourth floor in the you know there's little year a school panic as to see somebody with a his running some of the copernican serves as you can also find the age of it. a job advertisement but with the if you're interested in thank you very much work.
Feedback