ESM-TOOLS: a tool for Earth-System-Modellers
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Alternative Title |
| |
| Title of Series | ||
| Number of Parts | 60 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/42506 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
6
13
21
25
41
53
00:00
Physical systemEngineering physicsCAN busAdvanced Encryption StandardPhysical systemSoftwareProjective planeDynamical systemComputer simulationChannel capacitySheaf (mathematics)Computer configurationXML
00:38
Physical systemEngineering physicsInformationVideo gameBitDrop (liquid)InformationProjective planeComputer simulationPhysical systemLevel of measurementXML
01:27
Engineering physicsComputer simulationPhysical systemVelocityCompilerScripting languageInternet service providerSystem programmingIntegrated development environmentSupercomputerFunction (mathematics)outputGeneric programmingAcoustic couplerPhase transitionDifferent (Kate Ryan album)SimulationComputer simulationSimilarity (geometry)Moment (mathematics)Medical imagingCodeRevision controlVelocitySatelliteData structureMetrePhysical systemRun time (program lifecycle phase)Field (computer science)Different (Kate Ryan album)Scripting languageSimulationFunction (mathematics)MereologyBitProcess (computing)Modal logicConnectivity (graph theory)Virtual machineMultiplication signoutputAcoustic couplerIntegrated development environmentModule (mathematics)Scheduling (computing)InformationResultantSheaf (mathematics)WordOffice suiteCompilerFunctional (mathematics)Compilation albumCohen's kappaWater vaporQuicksortSpeciesDialectAddress spaceAreaCASE <Informatik>Internet forumAmenable groupSoftware testingSystem callXML
07:41
Function (mathematics)SimulationDifferent (Kate Ryan album)Phase transitionScripting languageMenu (computing)Engineering physicsSoftware testingSoftware developerGastropod shellComputer simulationIterationOrder (biology)Function (mathematics)Different (Kate Ryan album)BitDirectory serviceProcess (computing)Functional (mathematics)outputSoftware developerFrequencyRepository (publishing)Computer simulationMultiplication signCASE <Informatik>Software testingProjective planeState of matterMoment (mathematics)Scheduling (computing)SimulationScripting languageServer (computing)Variable (mathematics)InformationAuthorizationSimilarity (geometry)Cycle (graph theory)SupercomputerPolygon meshLattice (order)Continuous integrationStability theoryPhysical systemBranch (computer science)CollaborationismData managementCore dumpCodeFront and back endsSlide ruleLine (geometry)Computer fileGastropod shellRevision controlData structureVirtual machineArithmetic progressionModal logicMatching (graph theory)Binary fileMassInternet forumNeuroinformatikFood energyIntercept theoremProduct (business)HypermediaShared memory1 (number)WordMaxima and minimaRight angleSpacetimeLipschitz-StetigkeitMereologyQuicksortGame controllerComplex (psychology)Level (video gaming)DiagramProgram flowchart
17:08
InformationRepository (publishing)Projective planeMereologyAreaInformationSoftware developerImplementationFeedbackCuboidWeb pageFerry CorstenNewsletterTerm (mathematics)Vulnerability (computing)Key (cryptography)WordMenu (computing)Branch (computer science)State of matterBenutzerhandbuchWiki2 (number)FAQXML
18:55
NewsletterInformationNewsletterYouTubeTwitterInformationCore dumpMultiplication signXML
19:27
Meta elementPhysical systemComputer simulationProjective planeState of matterUMLComputer animationLecture/Conference
20:18
Projective planeDynamical systemCASE <Informatik>Physical systemComputer simulationEstimatorScripting languageThermal radiationWeightLaptopDifferent (Kate Ryan album)Software developerInternet forumSimulationSoftwareFigurate numberSinc functionTerm (mathematics)Connectivity (graph theory)Field (computer science)Latent heatBitRepresentation (politics)Expert systemVirtual machineExecution unitProfil (magazine)NeuroinformatikProcess (computing)Theory of relativityChannel capacityMereologyScaling (geometry)Mathematical modelCollaborationismCodeComputer animationLecture/Conference
Transcript: English(auto-generated)
00:00
for this interruption. So yes, I'm working at the Avian Bremerhafen in the climate dynamics section. And I also want to acknowledge the ESM project that I'm funded by. So ESM stands for Earth System Modeling
00:21
and the project that I'm working for is Earth System Modeling Capacity. So we're developing a software tool, it's called ESM Tools, and it's supporting system modelers to make their life drop a little bit easier.
00:48
So I'll start with introduction, what the ESM tools are, and then how we developed the tools and how we support users and how users can contribute to this project.
01:03
And at the end I will show some general information, how you can get information and contact us if you like. So what are the ESM tools? As I said, the ESM stands for Earth System Modeling or Earth System Models, and it's a tool that support the usage
01:24
of climate models or earth system models. So here's an example of actually an ocean model. It's not an earth system model, but it's a model about an earth system component. And in this case, it's the ocean.
01:43
So it's also an ocean model that is developed at Avi and Brimhafen in our group. And so it's called FESOM. And here you see results from a simulation done with the FESOM model.
02:01
So it's not in satellite image actually, and showing the velocity fields at 100 meter steps. And you see very nicely these different dynamical structures in the output. But I'm not, we're not talk about the model itself, but what comes with it to run such,
02:24
to carry out such a model experiments that you see here. So that was the purpose of developing these ESM tools. So, I mean, this is not the only model you can use.
02:43
It's just for the ocean, for example, but there are no different models for atmospheric dynamics, atmospheric chemistry, and all different models that belong to the ocean, to the earth system. And also a coupled version of these. So that's then called a system model
03:02
when these components are coupled. And so all of these models, of course they're different and they come with their own tools or scripts, how you, for example, download or compile these models.
03:21
And I mean, you're all written in Fortran code. And they all have different scripts, how you execute or run such simulation experiments. So, and this is the purpose actually of these tools, to make this process a little bit easier,
03:44
that you don't have to deal with all these different compile scripts and run scripts. So, and there we address some aspects of doing model experiments or simulation in this field.
04:02
So, related to the model themselves or the coupled earth system, earth model systems, and for the different HPC that you want to run these experiments on, and the different ways how to set up such an experiment.
04:22
And then it's all related to the data that you need to do such runs, that's input data or forcing data that needs doing run time and how you organize the output. So, therefore, we, so the SM tools are actually three tools.
04:41
One is related to the model and how you download them, how you configure if needed, how you compile it. And the second one is concerning the SM environment. So, on which machine you want to run your simulations.
05:02
And it takes account for all the different modules you have to load to also compile, have to compile environment, and for running such an experiment. And the biggest part of the tools are actually the runs, the SM run scripts.
05:22
And at the moment it's, there are shared functions and that provide you to have similar run scripts for all these different kinds of models that we support at the moment.
05:40
So, yeah, at the moment we support 10 different models. They include four ocean models, two atmospheric models, an ice sheet model, and bio-geochemistry model, and a glacial isostatic adjustment model. And also a different coupled system. So then we come to climate model
06:01
when you couple atmosphere to ocean, for example. And we have support for one coupler that actually takes, or is doing the exchange of the data between two model components, for example. And we support at the moment C6, so HPC systems.
06:26
So that's basically a introduction to what the tools are. And I want to say a little bit more about the biggest part of the tools,
06:40
which actually the user is interacting most with. And this is the SM run script. So it actually takes care of different steps doing such a model simulation. So at the beginning, you need to gather information
07:04
where you want to start your simulation from. So from what kind of input data you want to start, begin your simulation from here at the top. And then it's preparing the folder where the data are stored during the run time,
07:21
and then modifying them if necessary. And put all the data into a work folder where they are accessible during run time of your simulation. And then it actually submits the run to the job scheduler on the HPC that you choose. And when the run is finished,
07:43
it will also move the data to an output folder and will generate a kind of folder structure that is organized all the data in a nice way. And mostly you do these runs not in a hole,
08:03
but in different steps or runs. And if necessary, if you want to continue your simulation, then it actually resubmits it to the job scheduler. And then the cycle will continue again.
08:23
So now the idea is then to have similar run scripts for different models that you know don't have to deal with all these different scripts anymore. So here's an example of a very, so the least lines you have to put in the run script.
08:46
And this is also an example for the FASOM model that I showed at the beginning. So it's actually just a text.
09:01
You state different variables in this file. You say where your functions, the run script functions are actually lying on your system. And then you define on which machine you want to run your simulation. This is OLLI, this is the HPC at Aave in Bremerhaven.
09:24
And then you say what model you want to run, how much compute time it will take, and for which time period you want to do your simulation and then followed by a few variables
09:41
where the input data can be found and where your, actually the experiment should go to which folder. And since FASOM, it's running on unstructured mesh you have to define where it finds the mesh and how long these runs should be.
10:01
And then that's basically it. And then you execute this file and actually what's executed, it's not showing on this slide here. It's a little bit further down. So that's the minimal example that you need to do to do an experiment.
10:23
And the functions behind this will then take care of everything that you need. They copy the input data to your working directory and takes care of all the forcing data available. And then after the run it will store your output
10:42
and doing a post-processing if you want. So now I want to come to how we developed this tool at Aave and there's a timeline when this started in early 2017.
11:02
So it's actually the idea of the colleague of mine, Dirk Barbee, he came up with the idea to make all these scripts a little bit easier because he had to take care of all the models that were available or in use.
11:21
And then in May 2017 there was this ESM project started and it fits quite nicely into this project. And then in August 2017 I actually started
11:42
to work on this project and in the ESM project and then they were presenting these tools to the ESM project community. And yeah, it actually was quite successful.
12:01
So a lot of people were interesting in this idea. And then we had a first release in September 2018. So then it was a state where a lot of actually people at the Aave Institute were using or starting to use
12:22
these tools for their model simulations. And yeah, since then a lot of other models, so it was extend the tools to other models and so it grew a lot and we had a second release
12:42
just recently in March this year. So and since then at that time there were 10 models supported on six different HPC systems. So we have 30 authors that are contributing to this project.
13:01
And a third release is planned again for September this year. And a little bit outlook. So at the moment this is a collection of shared scripts and we already start to port the functionality in the backend to Python code.
13:25
And so this is actually the core developing team. This is, yeah, so Dirk and me and Louisa is the project manager of the ESM project. And I would like to acknowledge a whole bunch of people here.
13:40
They already contributed to this project and there are a lot of names missing here actually. So a lot of people that work at Aave but also on different institutions mostly that are also included in the ESM project. So here's a map. So the main development is located here
14:00
in St. Grima-Hafen and we have collaboration or contributions from Kie, also from here from Kotsdam, Aave and DSSET and DLA and Oberpfaffenhofen. And yeah, so we support,
14:22
these machines, that is the HPC at Aave and Grima-Hafen, it's OLLI, Mistral at DKZ and also in Jules and Jureka in Jülich. So as I said, it's based on Make.
14:42
So the ESM master tool is based on Make syntax and most of the functions are still in shell but will be ported to Python hopefully very shortly. And of course we use version control, in this case also we use Git. It's hosted on a GitLab server at the DKZ
15:02
so that's the German Climate Computing Center in Hamburg. We use a branching model. So we have a stable release branch and a developed branch where we test new developments and new developments are basically developed
15:23
in future branches. We have regular releases as you already saw for now twice a year. And yeah, we don't have such a continuous integration
15:42
at the moment but we have some testing but it's not, so this is very much work in progress and should be improved. So at the moment there we have one kind of test that is checking if the repository is updated,
16:03
if there's something new and then it runs one test simulation and if this is, if this passed then everything is okay but this is only one test for one snapshot because it's only one simulation that we can test. So this, we would really like to
16:24
further improve and make different kinds of tests with this. So we have weekly project meetings basically in the ESM project at RV and also for all the developer that are in RV that contribute to this project. We meet monthly that they have the possibility
16:43
to come up with new feature that they need or get information if something is new that they might be interested in it. And actually all users can contribute to this project because the way you get the tools is that you have actually your local repository
17:03
and you can use Git and contribute if you want. So how we do user support and how users can contribute to this project. So we have actually a user manual
17:21
and we use the GitLab Wiki so there's some kind of how do I area where you can state questions or FAQs. We do, now we did already the second user workshop for people that want to start using the tools.
17:44
The next one will be actually next week and we have a monthly newsletter. So how can the users contribute? And it's not necessary that they develop new features. They can participate with the documentation.
18:02
It's its own repository, actually its own project. Or you can contribute to the Wiki page and in the repository and yeah, maybe most important is that they are really encouraged to give feedback about the tools
18:22
if they find any bugs. It's really nice to get this information if there's some features missing they really need. So they can all use the issue tracker on GitLab. And so the last thing is they are all welcome
18:43
to develop new features which are then can be merged into the main development branch. So and with this I have general information how you, if you're interested, get information.
19:00
There's a webpage. Yeah, you can contact us, Dirk or me. And we have a Twitter account for this tool and actually also a YouTube channel. So Dirk made a few YouTube videos explaining how to use these tools. There's a newsletter that you can subscribe to
19:22
if you're interested. Yeah, and that's it. Thank you for your attention and. Thank you Nadine. Questions?
19:53
So yeah, sorry, I have to repeat the question. So it's about what is the status? It is a meta built system.
20:02
For existing tools. So for existing models, so.
20:21
Seems there are other meta projects existing. So why set up a new one? Which, is it, what do you mean? Okay, I don't know what of this tool actually.
20:41
So seems that needs discussion amongst experts. I suppose if you have such a big project, you got some funding, you got some assessment before. People got quite convinced that they need this tool. But the ESM project is not funding only this tool.
21:03
So it's just a part of the ESM project. Yeah, but I would have asked a similar question. The relation of this ESM tool to what is below is not, it was not perfectly clear to me. So what does ESM stand for?
21:24
It's Earth System. So ESM is just Earth System Modeling, or Earth System Models. And that's one software, or that's already a collaboration of software developers with different models. So an Earth System Model is,
21:40
where you have models for different Earth System components, like the ocean model that I showed, figure off. And if you have a coupled system, so you combine like ocean model with an atmospheric model, then you have more components of the Earth System.
22:02
So the more components you add to your, some models and coupled system, then you will come up with. Maybe I have to ask a more basic question. Maybe there are more outsiders like me here. Maybe we should first understand what you understand in your field by the term model.
22:24
So it's a simulation software that simulates. It's a simulation code. Yeah. And that includes a representation of the Earth's surface, of the ocean depth profile.
22:40
Yeah, so that's the job of the ocean model. So it's a simulation model about ocean dynamics, for example, and you can have different models, like atmospheric models for the dynamics or for the chemistry. And the phase-on is an ocean dynamic model.
23:01
And these existed since decades. Yeah, so I don't know exactly when FITM was developed, but yeah, simulation models that simulate the Earth System or components of the Earth System are existing for many years. So that's- And then you added this new layer on top
23:22
to make it more usable. Yeah, because they all use their different run scripts to execute the experiment. So that's running such a model on a HPC, for example, and to actually submit these kind of experiments
23:43
to the HPC that we try to make a little bit more convenient for the users of such simulation models like an ocean model. One more specific question about-
24:15
So a user can run these simulations on their own desktop
24:21
or it's always to be submitted to a cluster, a big machine somewhere? No, you do this on the HPC. So in this case, you have the model on the HPC system and the tools that we developed, you can also copy to the HPC
24:44
and then you log into, so you have to actually be on the HPC cluster, for example. You just cannot do productive research on a smaller scale. To do something in that field, you need the big computing capacity. For the global models, I would say yes.
25:03
I don't know if there are any that you can run on your laptop, but in this case, also the SM tools that you cannot have these tools on your laptop and then it will submit to the HPC. So the idea is to have this installed
25:22
basically on the HPC system and then you run your run script and then it will start the model that you choose, like the ocean model, for example. Yeah.