Bringing together open source scientific software development for HPC and beginners
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 287 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/56826 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
SupercomputerData analysisComputerSoftwareOpen sourceSoftwareOpen sourceSoftware developerComputer animation
00:24
ComputerMachine visionFeedbackSoftwareSoftware developerProcess (computing)Field (computer science)Moment (mathematics)Knowledge baseMultiplication signForcing (mathematics)Term (mathematics)State of matterWave packetMechanism designExpert systemSoftware testingFeedbackMachine visionService (economics)Group actionAreaForm (programming)Different (Kate Ryan album)MultilaterationAnalytic continuationCircleContent (media)SoftwareNumerical analysisOpen setDilution (equation)Universe (mathematics)Prisoner's dilemmaSpacetimeVotingConfidence intervalWater vaporPresentation of a groupComputational scienceHydraulic motorComputer animation
04:00
ComputerCodeoutputRight angleResultantKnowledge baseCodeSoftware developerMachine visionSlide ruleSoftware testingDataflowDemosceneData structureWorkloadMultiplication signView (database)Computer animation
05:17
ComputerRevision controlControl flowPhysical systemSoftware testingVisualization (computer graphics)Source codeRepository (publishing)Digital object identifierPrincipal ideal domainUniform resource locatorRepository (publishing)Lie groupSoftware developerDataflowKnowledge baseFeedbackResultantPoint (geometry)Computer fileMatching (graph theory)Set (mathematics)Term (mathematics)CodeSlide ruleData compressionTorusGame controllerElectronic mailing listRevision controlSoftware testingPhysical systemPerformance appraisalCondensationGraph (mathematics)outputComputer configurationCASE <Informatik>AnalogyMeasurementStaff (military)Link (knot theory)SupercomputerWechselseitige InformationRemote procedure callVisualization (computer graphics)Principal ideal domainMetadataMultiplication signWordDigital object identifierTest-driven developmentMathematical analysisComputer animation
09:46
Latent heatComputerKeyboard shortcutTerm (mathematics)Arithmetic meanSemiconductor memoryDirection (geometry)Game controllerWordPresentation of a groupForm (programming)Content (media)Point (geometry)Bit rateNamespaceAddress spaceCursor (computers)Electronic program guideGroup actionNormal (geometry)Context awarenessSystem callRoutingDescriptive statisticsKey (cryptography)MereologyImplementationKnowledge baseRoundness (object)InformationSeries (mathematics)Sheaf (mathematics)Range (statistics)Projective planeMaxima and minimaCategory of beingGoodness of fitEntropiecodierung1 (number)FingerprintLatent heatNumerical taxonomyMeasurementFile archiverRevision controlComputer animation
14:12
Visualization (computer graphics)Physical systemControl flowRevision controlSoftware testingComputerOpen setFreewareTerm (mathematics)Electronic visual displayInformationElectronic program guideComputer configurationBlack boxRight angleKnowledge baseDescriptive statisticsTerm (mathematics)Numerical taxonomyContent (media)AreaCuboidPoint (geometry)State of matterInformationVariety (linguistics)XMLComputer animationProgram flowchart
15:29
ComputerFeedbackKnowledge baseCodeFeedbackVideoconferencingPoint (geometry)Projective planeArithmetic progressionMeasurementGroup actionLink (knot theory)Revision controlElectronic program guideSoftware developerQR codePeer-to-peerGame controllerMereologySlide ruleQueue (abstract data type)Right angleComputer animation
17:36
Sheaf (mathematics)Data structureArmOpen sourceKnowledge baseSlide rulePhysical lawComputer animationMeeting/Interview
18:12
CASE <Informatik>MereologySlide ruleWebsiteComputer fileWeb pageOpen sourceLink (knot theory)Instance (computer science)Density of statesMeeting/Interview
18:57
Different (Kate Ryan album)WordReading (process)Computer scienceField (computer science)Meeting/Interview
19:57
Pell's equationProjective planeLink (knot theory)SupercomputerWebsiteFocus (optics)MereologyContent (media)Computer scienceParameter (computer programming)Observational studySlide ruleWritingLibrary (computing)QuicksortSheaf (mathematics)Physical systemAreaWorkloadRight angleDataflowLabour Party (Malta)Sign (mathematics)CodeMeeting/Interview
22:58
SupercomputerResultantMeeting/Interview
23:22
Scripting languageSoftware testingShared memoryProjective planeObservational studyOpen sourceMereologyInstance (computer science)Single-precision floating-point formatProcess (computing)Data compressionTerm (mathematics)Cartesian coordinate systemoutputLink (knot theory)Computer fileMeeting/Interview
24:06
MereologyKnowledge baseUniverse (mathematics)Ocean currentArithmetic progressionExtension (kinesiology)BitForm (programming)FrictionLatent heatFlagOpen setRight angleMathematicsDataflowSoftwareBuildingMaxima and minimaQuicksortMeeting/Interview
26:24
SoftwareForm (programming)Open setMereologyRight angleInstallation artGroup actionFile formatAbsolute valueMeeting/Interview
27:10
CodeRepository (publishing)TorusRule of inferenceControl flowMeeting/Interview
27:34
LengthRepository (publishing)MereologyCode2 (number)WorkloadPerspective (visual)CASE <Informatik>Group actionMeeting/Interview
28:24
Meeting/InterviewComputer animation
Transcript: English(auto-generated)
00:08
Welcome to our talk bringing together open source scientific software development for HPC and beginners. I will talk to you about how we are trying to help beginners to get going in the world of open source scientific software development.
00:24
Let me give you a brief overview over this talk. First, I want to tell you what we actually did. Afterwards, I am going to explain to you who we are and what our vision is. Thereafter, I will provide an overview over the workflow we propose.
00:43
This will be followed by one topic in more detail. After knowing our vision and the workflow, we present the feedback we got. In the next step, I will present our solutions to the issues raised in the feedback.
01:01
Here we are going to take a look at an example to illustrate our solutions. Towards the end, we will tell you what we are currently working on and what we have planned next. At the end of the presentation, I am going to wrap up the knowledge base features.
01:21
And after the presentation, there will be ample time for discussion. In the current talk, I want to talk about how we transformed an unstructured article collection into a knowledge base that we believe to actually offer value. Please note that in the following the term knowledge base will be abbreviated with KB on the slides.
01:44
We started this knowledge base as we have seen a demand for expertise and training to be provided, especially to every new member of a scientific welcome group. With continuous feedback from members of engineering working groups, we took the knowledge of the
02:01
CRC 1194 and enhanced and expanded the content with different improvements on the formerly bare articles. I will cover the improvements in more detail later. But let me talk first about who we are. I already mentioned the CRC 1194, which is a working group at the Technical University of Darmstadt
02:27
that is concerned with the development of numerical methods and transport and meting processes in open foam. They operate in the fields of software development and engineering.
02:40
They actually also do physical testing, but we will leave that out at the moment. Next to the CRC, you can see the Institute for Water and Hydraulic Engineering. They are engineers and we have a test user there with little to no experience in software development, what is ideal for our purposes.
03:04
On the other side of the CRC, there's the Institute for Scientific Computing. Here we have, except me because I am a mechanical engineer, experts in software development for HPC. My co-speaker Jan-Patrick Lehr, who will join us afterwards in the discussion, is also a member of this entity.
03:28
Next, you see the two initiatives, NHR forces and NFDI for Inc., that are driven by the German state to foster the development of tools and services to make research more sustainable.
03:43
These three areas of expertise form the circle of competence for our knowledge base. There are other institutes who have contributed to the knowledge base, but for the sake of simplicity, I focused on those with the highest impact so far.
04:00
Back to the sustainability initiatives. Tools and a structured workflow can lead to sustainable research. That is, imagine an engineer who maybe is inexperienced in software development or sustainability issues. He might be doing some research or development and actually come up with a great solution, but we
04:22
as his colleagues or any other person will likely not be able to access, understand or reproduce his results. Now I ask you, what worth has this research? On the other hand, if an engineer uses the right tools and workflow that
04:40
ensure reproducibility, that colleague should be able to understand and reproduce what has been done. Also, these methods oftentimes help with the quality of data, code and methods. Our vision with the knowledge base is to help enabling engineers or any other beginner to code development to do sustainable research.
05:06
Now, to help them do this, we wrote the knowledge base. On the next slide, let me show you how the workflow developed in the CRC looks like. This is the workflow the CRC has developed. You too could use it, maybe you would need some adaptations.
05:26
Let me introduce the workflow to you in a few words. The developer has a local repository and from there, pushes his code to the main repository. Hence, here we have the topic version control. A commit or a merge request then triggers a CI pipeline or adds CI and test-driven development to the workflow's features.
05:51
The CI pipeline again uses build systems, runs on an HPC with a container and when the tests have run, their results are automatically quantified and visualized.
06:05
These are the next three topics within the workflow. If the pipeline and the tests succeed, the new code can be cross-linked with articles and datasets. In the next slide, I will exemplary show you this workflow of cross-linking in
06:22
more detail since I assume most of you are very familiar with the other topics. So let me explain to you how articles, datasets and source code can be concatenated with what we refer to as cross-linking. At this point, you already have a remote Git repository. So in the first step, you create a
06:45
Git tag on the version of the code which you use on your data and in your article. Then you upload all the data you are able to publish to a data repository. That could be Zenodo or in our case it is Tehu DataLib, a repository hosted by the Tehu Darmstadt.
07:07
Ideally, the files to include in the upload are all the files necessary to recompute your results like input files and possibly your code on a container with the code and your results themselves and the evaluation of the results.
07:26
These evaluations typically would be further analysis and condensed data like graphs and charts. From your data repository, you will receive a PID. In the following step, you reference this PID as well as the Git tag in your article.
07:44
Every article gets a PID, often times a DOI. This PID is integrated into the metadata of the data repository and the metadata of the code repository in this last step. The mutual PID of data repository and of the code repository are also included.
08:06
What you get at the end is an article that is linked to all the data related to it as well as to the code. In the code repository on the other end, you can find the links to all articles and data associated with the code.
08:21
The data repository is linked to both other entities' analogies. As I have mentioned in the beginning, we did not start from nothing, but from a loose article collection of the CRC 1194 and a paper describing the workflow, I just presented you.
08:41
Our first steps were to put them on the internet with little to no modifications. We also reached out to seek feedback. Here is a list of the probably most often and most earnest comments we have gotten so far. The article collection was unstructured, had no or not enough introduction into the topics.
09:05
It lacked explanations in every regard, terms or vocabulary, goals, steps in the workflow and so on. It was incomplete and lastly, many articles were too specific for anybody outside the
09:23
CRC or otherwise very familiar with the tools and methods of the CRC's staff. So it was clear, we needed to add value to the articles and create what we now call knowledge base. Let me tell you on the next slide how we approached the feedback and what our solutions to those issues are.
09:46
To overcome the issues related to the shortages of the articles from the CRC, we took different measures from which I want to present six to you. Please do not get me wrong, the articles from the CRC were great in their own way,
10:03
but mostly were written as memory aids for the highly skilled and experienced people inside the CRC. They had been merely suited as introductionary guides and tutorials for beginners. However, without them we would not be where we are today.
10:21
Firstly, we revised the old articles. The measures taken in the first round range from restructuring and reformatting to adding links and explanations. We also did, and still regularly do, add articles expanding the topics. A special kind of article we introduced were guides. We used them to explain the overarching workflow and goals of each of our specific topics.
10:50
At the end of each guide, there is a section called where to start, in which we link our articles and guides to what we think of a good point of entry as well as to literature.
11:03
This brings me right to our next point. We added literature. The literature includes guides, handbooks, examples and tutorials. It is important to understand that we cannot possibly cover everything up to date from Git, GitLab, C++, Python and so on and so forth.
11:27
There are many projects out there dealing with one of our tools, including of course the official ones. Special to our knowledge base is that we bring together many tools in this unique way and can show how to combine them.
11:45
A major improvement for beginners certainly is our glossary, where we define the meanings of words or phrases in the context we use it. We do this in a rather short manner and we believe that our descriptions are well understandable for beginners whilst being comprehensive.
12:04
The great thing about the implementation in HugoDo is that we use short codes to mark the word in each article. They are implemented so that when hovering over the term with the cursor, a one sentence definition is displayed to the user.
12:22
If the user clicks on the term, he is taken to the full glossary article about this term. Another feature we added are taxonomies. We use three different ones, being tags, chapters and workflow. The tags are just normal tags with keywords like GitLab, OpenFoam or archive.
12:46
They are not part of a controlled namespace. Other than that, the chapters taxonomy refers to the topics of the workflow as I presented to you earlier. It includes, for example, version control, CI and cross linking.
13:05
The workflow taxonomy so far only distinguishes between the workflow in full flavor and in minimum flavor. The last one containing only the most essential topics in a reduced way. This helps to quickly determine the relevance of a given article when starting out on these topics.
13:28
With these improvements, we believe to have created a knowledge base that is well structured, easy to understand and rather complete. Each of these properties we try to maximize in an ongoing effort.
13:42
Another thing we do see in the knowledge base is that it combines broad general information with very specific ones. Arching from the goals of, let's say, version control to specific command syntaxes. Overall, we aimed to create content that is well suited for beginners.
14:05
And with the current and ongoing adaptions, we hope to have a high achievement rate at that. So let us have a look at the current status of the knowledge base. Each chapter contained a variety of articles.
14:21
But since the chapter is actually a taxonomy, you will also find the guide which I have marked with purple box and literature which is marked with a black box contained. But let us take a brief look at an article we are working on as of today. Note that this is a pre-recorded talk.
14:43
In the top area, you can see the three colored taxonomies which I have been talking about. You can also see some of the formatting options Hugo has given to us. One example of these are such info boxes like this orange one.
15:02
On the next slide, I would like to show you how the glossary tooltips are working. When you hover over a term like displayed here, a very short description on the term in question will be displayed to you. You can click on the term to get to the actual glossary with much more detailed information.
15:25
We believe this to be a very powerful feature for every beginner. Let me now show you what we have planned next. We keep working on the knowledge base. As I have mentioned, we already added guides, a glossary and taxonomies.
15:42
These are overall done points. Of course, they get updated as we move along. Measures in progress are revision and peer reviewing of articles for further improvements as well as adding articles and literature. These three measures are probably always going to be items in action.
16:05
Lastly, but not least on this board, we are currently thinking about how we can restructure the articles. On the next board, there are also ideas we would like to engage but haven't started with yet. These are adding interactive tutorials, collecting more feedback from new and unpracticed beginners
16:27
and also adding introductory videos to our guides to improve our didactic offering. Now on this last slide, let me sum up the most important points.
16:40
The knowledge base we created and the practices therein help engineers to do well with their projects. It helps them to write better code and to do retraceable research or code development. The topics featured include version control, cross-linking, CI and so on and so forth.
17:03
But the knowledge base itself not only is peer reviewed but also features, for example guides, a glossary, taxonomies, literature and so on. With this, I would like to come to an end and of course not without displaying the link
17:21
and the QR code to our knowledge base, where you can find everything I have been talking about. Now I am looking forward to your questions.
17:49
Thanks Moritz for this talk. We can start now with the Q&A. So we've got a question about how is the data from all those external sources
18:03
and parsed and structured into the knowledge base? Is that automatized? Is that manual work? So great question. If I get it right, this refers mainly to the literature section we have, where we link external sources like man pages or slides from other persons.
18:28
So what we do is either we have access to those slides or files and then we can attach them as part of the actual page as a file.
18:43
Or we just link to the external site and refer to it in the case where there is the manual, the documentation. Okay. And for instance, you were talking about this feature where you can highlight some keywords
19:01
and just add definitions to them. I guess that's done by somebody. So that's manual work. Yeah, that's actually manual work. And the reason for that, maybe we realized when we were talking to some of the engineers that read the articles, they said, oh, we struggle sometimes when we look at
19:23
the documentation for a feature that we're using because we don't understand all these other words that come up. So trying to understand one feature requires me to understand 10 other words. And so this is why we try to reduce the wording to as much normal words as possible.
19:42
Use potentially metaphors that work for engineers, depending on obviously which field they are in, but keep it less computer science-y and more geared towards people that are, I just want to get this to work. Exactly. So more step-by-step and practical guidelines. Yeah, exactly.
20:02
Okay. So next question is by Chris saying, does this project link with HPC Carpentry? That's a very good question. I'm going to steal this one from Moritz just for now, because that came up in another discussion that we had.
20:20
And I've only heard HPC Carpentry and I wasn't fully aware of their content. However, I looked it up in the meantime. And from what I see from HPC Carpentry is their focus is more on, let's say, how do I use the system or how do I write HPC Python code? How do I get these Python libraries to run performant on my system?
20:43
And please correct me if I'm wrong because I've only dug like so far into it. For us, it's more, if you don't know how to approach computational sciences in a way that, how do I keep my data around? Like, how do I know what I did like six weeks ago when I ran these experiments?
21:04
How do I keep the parameter studies around? Basically, so that's why we started with a workflow. And this is actually done by Tomislav Maric, who is I think credited somewhere on the slides during the CRC that we mentioned. So it's really more of giving those engineers, those people that just now have to use computational sciences, a reasonable reference workflow,
21:29
how to do their science in a structured, in a maintainable, in a reasonable way, sort of. And along this workflow, they need to use tools.
21:41
And then we try to explain them how to use these tools to get to this workflow. So it's not, oh, by the way, this is how you do Python for HPC. We're not trying to do this. It's more like if you have to do computational experiments, you can follow this workflow to maintain an overview of your data,
22:02
to know which data went into what paper and the other way around. And so you get this, this is also why we have this cross-linking section. So it's really more about doing the science and then having articles, for starters, articles like how you do this. And then link to other resources if you want to go further down that road and do more in that area.
22:23
Then you can go out to the external sites that actually teach you how to do, I don't know, HPC Python or some other stuff. Yeah, so very good question. HPC Carpentry is, as far as I know, more on that other part that if you need to use it, you need to go down that rabbit hole and implement some very hefty HPC Python or Fortran or Julia or whatever you have, right?
22:50
Then you look there. And for us, it's more of the how do I approach this at all? Like I'm lost, right? Yeah, totally agree. I also see that the workflows in HPC become more and more complex.
23:04
And that can be overwhelming for many researchers. Like they don't only have to do the simulations, they have to handle the data. And also handle how the results are going to be stored and then make available to follow up research.
23:20
Yeah, exactly. Exactly. There can be a lot of important that I share this concern. And that links actually to my question next. I also saw that input files and test scripts are part of this workflow that you define for your researchers. So what's the source of those?
23:41
Because for instance, in terms of what you call source code or I guess it's input data or job script or something like that, that can be quite variable. So I guess that the researchers are also modifying those. And for the test scripts, I also guess that can change a lot from one application to another or from one project to another.
24:02
So how do you handle that? Yeah. Robert, should I jump in or do you want to answer? If you want to go ahead, please. Okay. So maybe a little historical note on that. Because what most of the current examples, at least for the little more elaborate things, are also part of the CRC.
24:25
So they are open form based examples. How you would achieve something using open form or with open form. So they are somewhat tied to numerical libraries with open form. However, eventually what we want to have is, let's say, an easy to understand, very basic example how you would do this.
24:50
As I said, it's more about the workflow and how you fit the pieces together rather than explaining or teaching the pieces to the fullest extent. And so for now, we have these open form examples that would help for the specific situation.
25:07
And now we're working together with engineers from our university and other universities to get what are the problems, what are the things that people are interested in when they get started with that. What are the challenges that they face?
25:22
And then extend the current knowledge base to address these issues with basically a minimal running working example. So this is like if you want to follow this workflow, this is the minimal example that you can understand. Because we're not saying like, oh, by the way, you have to pass this weird flag
25:41
to CMake because of how open form builds stuff and then you have this and that. No, no, no. It's really just like do this and do this and do that. So that's the goal and we're working on that. So it's still a work in progress knowledge base. Obviously, it probably will always be work in progress because things change, right?
26:01
But yeah, so we are moving towards these more minimal examples to guide researchers and take them by the hand and show them what could be done. And then they can deep dive themselves into like whatever is available in their domain, whatever specific solver or tool they need.
26:21
Okay. And just a little bit of friction because you talk about CMake and stuff. So compiling and installing the software like open form is not part of this workflow, right? So I guess that the open form is already available somewhere and it's using the software. Yeah, honestly, I'm not completely sure about whether there's actually an article on how to compile open form.
26:45
It might be there just for like historical reasons that I just outlined. But it's not the overall goal to have for every possible solver package installations and how you get things running, right? No. That would be a little too large.
27:02
And then we would probably say, oh, use EasyBuild or use SPAC or whatever if you're having trouble with that. Yeah, absolutely. Okay. So we have one minute and a half maybe just for the last question. It seems to me that the base layer of a data and code repository is assumed here but not uniformly available.
27:23
Can you provide reference for how you built that layer? So I'm not completely sure if I understand that question actually. Yeah, probably we can discuss about that in the breakout room because we only have one minute left. Yeah, I'm sorry. Yeah, we can absolutely continue in the breakout room and discuss in length.
27:53
So, Alan, maybe, do you mean that you have, we assume that we actually have
28:00
a code repository and you have a data repository as the, from the researcher's perspective? If that is the case, we try to convince people to have that and to do that as it's part of the workflow to actually have repositories for both for code and for data such that you can actually then link them together.
28:23
Okay, there's only 20 seconds left. So let me thank the speaker and all the audience for your attention. And please, everybody interested, come into the breakout room. Thank you, Alex. Thank you.