Selecting a Finite Element Analysis Backend for Exascale Fusion Reactor Simulations
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 490 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/47253 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Front and back endsInfinityMathematical analysisElement (mathematics)SimulationPhysicsFinite element methodFood energyComputer simulationMultilaterationGoodness of fitFront and back endsDistribution (mathematics)Computer animation
01:33
Color confinementComputerFreewareField (computer science)Set (mathematics)Shape (magazine)Color confinementArithmetic progressionComputer animation
02:16
CollaborationismOrder (biology)Figurate numberFood energyBuildingComputerComputer animation
03:08
Mathematical analysisSimulationPhysical systemPhysicsPublic domainLibrary (computing)IterationTemporal logicElement (mathematics)InfinityCodeParallel computingSupercomputerIntegrated development environmentExtension (kinesiology)InternetworkingProcess (computing)Open sourceAsynchronous Transfer ModeMixed realitySoftware engineeringOpen setSoftware developerElectronic mailing listFormal languageOverhead (computing)Front and back endsFinite element methodGoodness of fitInternet forumSoftware engineeringWeightEmailPhysical systemSoftware developerOrder (biology)Interpreter (computing)Library (computing)Formal languageComputer architectureComputer simulationStapeldateiUniform resource locatorVirtual machine1 (number)Mixture modelVariety (linguistics)Shared memorySimulationSoftwareElectronic mailing listExterior algebraFluidOverhead (computing)Semiconductor memorySocial classProcess (computing)Object (grammar)NumberSet (mathematics)Spektrum <Mathematik>Food energyPhysicalismPoint cloudParallel computingProduct (business)Maxima and minimaInternetworkingRevision controlOpen sourceOpen setCodeComputer animation
09:59
Archaeological field surveyGaussian eliminationVector potentialSupercomputerSoftware developerVector potentialArchaeological field surveyLibrary (computing)Software1 (number)Beta functionAlpha (investment)Electronic mailing listComputer animation
10:27
Finite element methodInformationProcess (computing)Set (mathematics)Library (computing)ResultantDifferent (Kate Ryan album)Computer animation
10:54
Nichtlineares GleichungssystemPoisson processBitMultiplication signNichtlineares GleichungssystemExecution unitSquare numberCubeComputer animation
11:15
Polygon meshOpen sourceExecution unitCubePolygon meshResultantPairwise comparisonScalabilityComputer animation
11:51
Read-only memoryElement (mathematics)Level (video gaming)Semiconductor memoryLimit (category theory)LinearizationComputer animation
12:18
Finite element methodPortable communications deviceProcess (computing)WeightRankingSoftwareSupercomputerFile formatScalabilityOpen sourceGoodness of fitFirst-order logicPolygon meshBenchmarkFormal languageResultantInteractive televisionComputer fileComputer simulationPhysicalismScalabilityMathematicsUsabilityRankingMultiplication signBasis <Mathematik>Overhead (computing)Pairwise comparisonSet (mathematics)Order (biology)Level (video gaming)Virtual machineLibrary (computing)View (database)Software testingPareto distributionPoint (geometry)Line (geometry)WeightProcess (computing)2 (number)Software developerOpen sourceFile formatCurveSemiconductor memoryFront and back endsDifferent (Kate Ryan album)Nichtlineares GleichungssystemBitTerm (mathematics)MeasurementFinite element methodComputer animation
18:52
ComputerDiscrete groupTerm (mathematics)Numeral (linguistics)Library (computing)Nichtlineares GleichungssystemMereologyError messageSimulationPhysicalismSemiconductor memoryProcess (computing)Entire functionOverhead (computing)ScalabilityPublic domainInterpreter (computing)Context awarenessBefehlsprozessorQuicksortMultiplicationLatent heatComputer simulationDifferent (Kate Ryan album)Point (geometry)Default (computer science)ResultantPosition operatorCodeAuthorizationCodeBenchmarkFormal languageImplementationPartition (number theory)Military baseComputer animation
25:26
Point cloudOpen source
Transcript: English(auto-generated)
00:06
Okay, we're ready to start our next talk from Alexander on finite element analysis. So please welcome Alexander. Good afternoon, my name is Alexander Dubas and I'm from the United Kingdom Atomic Energy
00:25
Authority and I'm giving a talk on selecting a finite element analysis backend for exascale fusion reactor simulations. First of all, if you're not familiar, I'm going to give you a brief introduction to fusion and the challenges behind it.
00:45
Simply put, we take two isotopes of hydrogen, smash them together and we get some helium, a neutron and lots of energy out. So we take the two isotopes we use typically are deuterium and tritium.
01:03
Deuterium is a naturally occurring isotope that's in everybody here in all the seawater and it's thoroughly abundant on earth. Tritium is less abundant but with a fusion reactor that's well designed you can breed
01:21
your very own fuel inside your reactor. The energy distribution mostly comes out in this neutron here and I'll come back to that later. So it's about 80% of the energy. So how do you do fusion? Well you need to generate a plasma first to get your nuclei free of their electron
01:44
shielding and we do this by magnetically confining it in a set of magnetic fields which have a set of coils in a coil, poloidal and toroidal field coils which both confine and shape the plasma.
02:03
In most fusion reactor designs these are superconductors. It's debatable as to whether the high temperature superconductors will work or low temperature superconductors. But currently in progress being built in the south of France is ITER.
02:22
This one does use superconducting magnets and you can see the plasma chamber in the middle of this picture here. This is an international collaboration and its aim is to ultimately prove that we can get more energy out of fusion than we put in.
02:42
It roughly costs and it's not finished yet so there are no actual figures but it's in the order of 20 billion pounds. Now this is rather expensive so it would be nice if we could simulate this inside a computer and know it's going to work before we actually invest in building a large
03:04
building and spending a lot of money. So how do we simulate it? Well it turns out it's actually quite challenging. First of all we need to capture the energy from these fast neutrons so we need neutronic simulations and because it's magnetically confined it's electromagnetic
03:24
simulations. Obviously the magnets are going to maybe induce some structural deflections as well as the weight of the machine itself so that's going to deflect that and that's going to change your neutronic simulations. And you want generally to get the heat out of your system so we need thermal simulations
03:46
and then we don't want all the heat to stay in one place so we use fluid simulations to extract it and so this is a highly connected multi-physics problem. So there are two ways of approaching the multi-physics issue.
04:00
One is a single tightly coupled simulation so you have one single executable solving a large linear system with all the physics involved. Now this ensures that you're capturing all your strongly coupled phenomena but it's numerically challenging to build this and some say it's not even possible.
04:21
The alternative is you take your best-in-class solvers for each of your independent physics and you couple them with a third-party library. This is an easier method so you're guaranteed the best performance for each individual class of physics and it's easy to decouple the ones you don't need for your current simulation
04:40
but the temporal accuracy may suffer and you might not capture strongly coupled physics if you get it wrong. So the approach we're focusing on in our current work is looking at a single tightly coupled simulation using finite elements and so we need to select a back end to build
05:03
our finite elements simulation on. Now we need to select this finite element back end in an objective fashion rather than going oh I know how to use this one let's just use that because it's my favorite and my mate works on it.
05:22
So I'm presenting today our objective method which includes a lot of subjectivity but we'll get to that. So how do you define a set of objective criteria to benchmark your finite element libraries?
05:41
Well for one to simulate a fusion reactor is going to require somewhere in the order of an exascale machine. So it's an exascale simulation and it has to be designed as a parallel code from the outset. You're not going to reach the performance required without a parallel first approach
06:01
and therefore parallel first is our first criterion. The second criterion, so we need to be able to run on whatever computing resources we have available which could be anywhere in the world especially with the cloud including locations without internet access
06:24
on your batch nodes could be any number of processes. So you need to have one a license that will scale to any number of processes that you want to use and ideally you want to be able to permit extension and modification.
06:43
So naturally this would lead you to an open source license except not necessarily theoretically you could license software that is both permissive and proprietary but no one's that brave.
07:02
So another problem, there are no exascale machines yet so what does it look like? It could be a multitude of highly vectorized CPUs, it could be a mixture of open MP and MPI so shared and distributed memory paradigms, it could be solely GPU or heavily GPU.
07:25
So one of the criteria is to have a very portable library that supports a variety of architectures. One of the criteria also is that we want to be able to add to our backend if we need to
07:46
and so it has to be open to external contribution so that's permission and also we have to be able to add to it easily and not waste time so that generally requires that we follow good software engineering practices.
08:04
Finally, our main criterion is it has to be well supported including a good community around it, forums where questions are answered, mailing lists or IRC rooms, workshops and tutorials and good documentation both for developers and for users
08:25
which are slightly different. Now there are also some implicit criteria. One is that it's a compiled language that we need. Interpreted languages are getting very good but they still incur an overhead.
08:43
We did benchmark Phoenix against its C++ backend Dolphin and performance was nearly identical apart from an extra memory overhead of the Python interpreter and when you scaled it up to a million processes, you're then using 80 terabytes of memory just in Python processes.
09:10
The other implicit criteria is it's got to be production ready so to speak. It's got to have a stable API or at least the developers need to express that it's confident
09:21
so we set a criteria of having a minimum version number of one which is somewhat arbitrary but it's a way of ruling out libraries that aren't ready for use yet and the other criteria has to be actively supported, has to be actively developed
09:43
and so we excluded anything with no development in the past six months and you'd be surprised how many libraries there are, well you might not be surprised, how many libraries there are that are effectively in the finite element graveyard.
10:01
So our initial survey found 35 potential candidate libraries and we went through and eliminated those that weren't parallel first, the ones that were in early development or listed as alpha or beta that were poorly supported, inextensible both from permission or from poor software practices and those that were abandoned
10:24
and we came up with a short list of deal two which you might have seen a talk yesterday. Dune, Dolphin, LibMesh, Mfem, Moose and Nectar++.
10:42
Now obviously some subjective criteria were included and if you did this same process today you would get a slightly different set of results as more libraries have matured. So how do we measure the performance? Well I've levelled up from solving the Laplace equation on a unit square
11:02
and I'm solving the Poisson equation on a unit cube and to also measure when there's a little bit more CPU work the time dependent equation was also measured. The mesh was a unit cube generated in GMesh as having an open source mesh helps
11:27
but this led to one slight problem. We couldn't get the mesh into deal two which is rather unfortunate because it seems like a very good library and if you were in a talk yesterday the claims that it scales
11:41
and performs better than commercial packages is corroborated by our results but we can't present them in comparison because the same mesh wasn't used. We also measured the memory usage because that's quite important when you have a limited amount of memory and at this stage we had to eliminate NECT++
12:03
which uses rather a lot of memory partly because it didn't solve with piecewise linear elements so we had to use quadratic elements and thus we couldn't get the memory down to a reasonable level. On to the scaling.
12:22
We measured performance scaling up to a thousand cores. Obviously with strong scaling it dropped off as our problem size here and this seems to suggest that Dolphin is the best library, right? With Moose in second place and the rest below it. However, looking into this slightly more in-depth
12:42
it turns out a lot of time was spent using the mesh importer and we're not trying to measure how good the mesh importer is. We want to know how well it solves so we looked at the solver scaling instead and so this gave a different set of results with Dune appearing to scale very well
13:03
Mfem and Dolphin being roughly on a par and libmesh not scaling so great. Moose isn't included in these results because we couldn't get the directly comparable solver timings out of Moose.
13:22
However, it's not just about scalability it's also about how quick it is. So Dune seemed to scale very well but it took too long and it never reached the same speed level. So Mfem and libmesh were both really fast.
13:41
Dolphin and Moose scaled well but were a bit slower. Maybe they overtake further down, they don't. And so that's the performance side of things. Honorable mentions, I would say Mfem had a very, very nice build process. It took literally half an hour to set it up on whichever machine I wanted to
14:06
and Moose has a very nice multi-physics coupling designed to allow arbitrary physics coupling. So which one's the best? There wasn't a clear winner.
14:20
The various libraries seem to sit on a Pareto front between performance and usability. So to pick a winner, some advanced maths of essentially weighting their performance and their quality to try and give an objective ranking to their assessment
14:46
and we scored them on their quality on the basis of their installation. So how easy it was to build both on this laptop and on our supercomputer. How usable it was.
15:01
So how easy it was for a user to get the simulations that they want. Their documentation which is very important especially when you're trying to learn and evaluate a finite element library in a week. And we eventually came up with a few answers.
15:22
Obviously the best library for what you get depends how you weight these equations. So if you look at pure performance, for example if you're going to exascale performance really matters, Mfem would be the best one according to our results.
15:44
And when your simulations cost about a thousand pounds a minute, I think roughly at the exascale, your developer time becomes cheap again.
16:00
And having well documented, easy to use portable build processes is obviously very important. User interaction was important and being able to support common open formats so those that couldn't import the gmesh1 files were excluded. And good tutorials, examples and source comments.
16:24
I believe that's all I have to say. Any questions? So you discounted anything that's not compiled from the start
16:44
without I think much consideration as far as I can tell. And I wonder why that wasn't, if you had actually done any benchmarking? On which ones? Anything that was using Python for instance, anything that wasn't compiled.
17:01
So we did do some benchmarking. So you asked, we discounted anything in interpreted languages like Python and asked if did we do any benchmarking and we did. So we did measure Phoenix against Dolphin and in terms of scalability and performance, it was almost identical
17:21
because it caused a C++ backend. But there is, the memory overhead is the main difference there. You used a, this is like a first order mesh? Yes. Yeah, did you try anything with, I don't know how many of the solvers support higher order,
17:43
but did you try any higher order solves? Because that's really where mfem was designed to shine. It's for like curved meshes and stuff that would use your GPU better. We didn't try high order meshes in this because implementing it in the various different libraries becomes a lot more of an involved process,
18:03
which is also why we didn't look at more involved equations for every single library. But yes, I agree, mfem does shine there. Yeah, it's a bit in the same line. I mean, the thing you want to simulate is from a physics point of view
18:23
extremely challenging, as you said. And as a test case, you take two very, very, very simple things. Do you think that actually is even representative for the thing you actually want to simulate? Because they might shine very well in these simple problems and completely miserably fail in the thing you actually want to solve.
18:41
That is correct, and it is a potential flaw. However, without implementing a full multi-physics solver in every single back-end, we chose a standard benchmark that we could definitely implement in every library. And do you actually even think we have codes at this point
19:02
that can actually do that at exascale level? No. Because I would think you actually still, people are developing these things, and as far as I know, it does not yet exist. This is the start of a large package of work to develop the simulations we need. So in terms of multi-physics, we've sort of nailed bi-physics,
19:22
where you have two coupled physics, and we're starting on multi-physics, so three coupled physics domains are sort of coming in. There's nothing that I'm aware of that does everything we need to. And just as curiosity, what exactly do you want to simulate of the whole ETF thing? Do you want to simulate the entire reactor,
19:41
or do you have specific research questions? Ultimately, yes, we would like to simulate an entire reactor. But on the way there, there are many different components, such as the diverter and blankets, where you have a lot of physics interacting, but you don't have to scale up to the entire reactor size, and so you can use a smaller computer.
20:02
Preferably before it does its first run. Here we're dealing with multi-physics. Why only considering finite elements? In terms of building a single solver that can solve everything,
20:25
I think it's probably the best discretization. I don't have any evidence to back that up necessarily, it's conjecture, I agree, but it seems the best way to be able to implement all the physics that you need.
20:41
Whereas I would agree if you were to do, say, multiple coupled solvers, then you would look at other types of simulation. Hi, thank you for a very nice talk. Is the problem you're trying to solve sensitive to numerical accuracy or instability,
21:01
and did you try to assess those libraries on this problem? We looked at, so part of the reason for using the Poisson and time-dependent equations is that we could then have a known variable put in,
21:21
so we have a definite solution, so we can compare the error there. But in terms of further assessment, that's as far as we went. Thank you. Since you're really evaluating the long-term sustainability of your project,
21:41
why only looking into making everything in one solver and not go with the partition way, for example? Because the performance is not necessarily the answer here. That seemed like two questions, which would you like me to answer? Let's start from the first one.
22:00
Can you repeat the first one? Yes, so why are you really trying to pick one of the, while you have so different applications, why are you really trying to pick one of the libraries for everything and not follow the partition way, that then you could have sustainability in the long term? So the work I'm presenting here is based on looking at that one approach.
22:22
We are also looking into the coupled approach, so we are covering both bases there. So for the solver part, did you use just, so you used one solver for everything, but I mean, like, mFEM has lots of solver backends, right?
22:41
You can actually plug in for all those, for those different frameworks, different solvers that they're really designed to work with. So do you think your solver results would be better if you did that? Potentially, so this is one point that I'd possibly like to make in that we tended to follow what was recommended in tutorials
23:00
and available documentation for each package, so we picked whichever solver was recommended for it, which is more a question of, does the library set sensible defaults or advise sensible things for performance? Well, I was going to ask the same question,
23:21
but I have actually another one, and I would maybe just quickly say that, like, I think there are many problems where, like, you just cannot, like, the library cannot set the defaults because the problem is very specific and it requires domain knowledge. But my actual question is, like, I want to suggest to you again to maybe re-evaluate your position on non-compiled languages
23:45
because I, are you familiar with Fire Drake? Sorry? Are you familiar with Fire Drake? Oh, you've heard of Fire Drake, yes. Yeah, so I'm the original author of Fire Drake, and in my PhD, I actually did, so it's an API-compatible implementation of Dolphin or Phoenix,
24:04
and we generate code for the entire solver at runtime, and I benchmarked up to 1536 cores in my PhD thesis, which was, like, six years ago, and I was able to beat every other package, easy.
24:20
That's a bold claim. Well, every other package, well, I was, sorry, I was able to beat the Dolphin, also the C++ version, in terms of performance, and, like, I wasn't, we weren't running into memory issues, so, like, just as another nudge maybe.
24:41
Yeah, the memory issues is then a scaling thing, so if you're only running single core with only one interpreter, it's a tiny amount of memory, so it only really becomes a scaling problem when you have multiple processes. Again, this comes back to, we don't know what exascale looks like. If it's fewer processes with lots of GPU dispatch,
25:01
then perhaps the memory overhead of running an interpreter isn't an issue as much as it would be if you run fully on multiple processes on CPUs. Thanks. Sorry, we're out of time. We'll have to meet the speaker after. Thank you very much. Let's thank the speaker again.