Generation of a wrapper library for MPI - MeDiPack
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 60 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/42474 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
6
13
21
25
41
53
00:00
Code generationLibrary (computing)Library (computing)Wrapper (data mining)CodeSoftware frameworkTopological algebraXMLJSON
00:15
Operator (mathematics)Operator overloadingSinePrice indexInformation overloadOperator (mathematics)Type theoryStatisticsWorkstation <Musikinstrument>Bounded variationAutomatic differentiationPerformance appraisalCalculation
00:47
Derivation (linguistics)Military operationInformationData storage deviceLibrary (computing)Variable (mathematics)Point (geometry)Reverse engineeringSweep line algorithmComputerSinePrice indexSoftwareOperator (mathematics)Operator overloadingAsynchronous Transfer ModePerformance appraisalType theoryWrapper (data mining)Topological algebraFunction (mathematics)CoroutineExtension (kinesiology)Gamma functionImplementationCodeDirected setTopological algebraFeasibility studyCode generationRankingData bufferMessage passingScripting languageComputer fileoutputDisintegrationIntegrated development environmentDerivation (linguistics)CalculationImplementationInformation overloadCodeType theoryFunctional (mathematics)Library (computing)Message passingGraph (mathematics)Operator (mathematics)CuboidScripting languageTopological algebraSystem callInformationGradientCartesian coordinate systemDirection (geometry)Multiplication signSieve of EratosthenesComputer programmingReverse engineeringBitDifferent (Kate Ryan album)SummierbarkeitRepresentation (politics)Process (computing)Point (geometry)Extension (kinesiology)Data typeAsynchronous Transfer ModeProjective planeTap (transformer)Data storage deviceWorkstation <Musikinstrument>Variable (mathematics)CoprocessorLatent heatIntegerFunction (mathematics)SpacetimeCausalitySoftware testingCoroutineMatching (graph theory)Interface (computing)outputNeuroinformatikDisk read-and-write headAlgorithmBuffer solutionOrder (biology)Transformation (genetics)Wrapper (data mining)Right angleGoodness of fitLecture/ConferenceComputer animation
06:11
Scripting languageTopological algebraoutputComputer fileFunction (mathematics)CodeIntegrated development environmentDisintegrationContent (media)Code generationSystem callTopological algebraLimit (category theory)Data structureMathematicsTemplate (C++)Normal (geometry)Line (geometry)Asynchronous Transfer ModePrice indexData typeCountingString (computer science)BuildingMetropolitan area networkRevision controlComputer configurationSoftwareStatisticsEmailTelecommunicationLibrary (computing)Wrapper (data mining)AverageCode refactoringImplementationPerformance appraisalBlock (periodic table)Order (biology)Graph (mathematics)Asynchronous Transfer ModeFunction (mathematics)Phase transitionCodeElement (mathematics)MengenfunktionNeuroinformatikEndliche ModelltheorieTopological algebraPerformance appraisalAttribute grammarLibrary (computing)Topological algebraRight angleFunctional (mathematics)Data structureOffice suite1 (number)Computer programmingParameter (computer programming)Computer fileFormal languageLine (geometry)String (computer science)Operator (mathematics)MathematicsoutputStatisticsEmailSoftware testingOnline helpProcess (computing)TelecommunicationSpacetimeCategory of beingSound effectOverhead (computing)Point (geometry)WordMultiplication signArithmetic progressionSoftware maintenanceGraphical user interfaceData conversionProgrammschleifeCASE <Informatik>Loop (music)INTEGRALComplete metric spaceCore dumpReal numberTerm (mathematics)EmoticonGoodness of fitLatent heatCuboidPoisson-KlammerFlow separationSoftware bugRootError messageTemplate (C++)Block (periodic table)Information overloadContent (media)Integrated development environmentDivisorGroup actionXML
15:16
AdditionView (database)Topological algebraCodeText editorCASE <Informatik>Special functionsDifferent (Kate Ryan album)ImplementationComplete metric spaceTopological algebraTranslation (relic)Source codeLibrary (computing)Right angleLecture/Conference
Transcript: English(auto-generated)
00:00
So, first I will motivate why we need a wrapper library for MPI and what is AD. Then I will provide the code generation framework I used, will introduce the best practice workflow I have found out over the development, and then I will give some statistics about MediPack.
00:20
So, what is algorithmic differentiation, you might ask. First I want to show you how we implement it. So we implement algorithmic differentiation by creating our own calculation type, then we overload every operation, you can do this on this new operation type, like the sign plus and so on, we evaluate the prime evaluation and then we do other stuff.
00:45
And what do we want to do with other stuff? We want to compute derivatives and that's done by using our new calculation type, we exchange all floating point values in the application with this new type, and then we
01:01
store information for every operation. And then, over a reverse, with this stored data, we can compute the derivatives of what you have computed in your program. And now, what is computed? Basically we compute here what is called the reverse mode of algorithmic differentiation,
01:24
and that's just the Jacobian of F, F is your program, and O is the mathematical representation of your program, by your input variables, transposed, multiplied with some right hand
01:43
side, which is basically a direction you can give what you will evaluate this. So if you have just one output value here, you can give there a one and you will get the full gradient of your application that you have computed. I want to stress here that DF by DX is never set up explicitly, you compute everything
02:04
in the background. So that's AD, and now we want to couple this with MPI, so MPI is message passing interface, I expect that everybody knows about this, so I will not go into the details. And the AD type program needs a special treatment by MPI because of this integer I have shown
02:25
here previously, this cannot be sent directly to another processor, and we need to store the reversal of the MPI commands. So if you have here a simple example, two processors, one sends to the other processor, then in the reverse mode, the receive is actually then a send in the reverse mode,
02:45
and the send is then a receive in the reverse mode, so we need to store this and to handle the data, and that's why we need a wrapper library for MPI, because if somebody exchanges in his application, his calculation type is our AD type, and he uses MPI, then this should also work out of the box, and therefore the wrapper library.
03:04
So the buffers need to be modified, that is our one requirement, and the reversal of the MPI commands need to be stored, and we want to have a 100 percent treatment of the MPI library, because if somebody comes on and, yeah, I want to use your tool, okay, I'm handling with the MPI library, but I just can handle two functions, then
03:22
nobody will use it, so you have to say we can nearly support 100 percent of the library. So, and the general approach we want to do here is that we overload all MPI routines, we do this here by saying, okay, an MPI send needs to be modified into an AMPI send,
03:42
and we need to create extended data types, so an MPI request gets an AMPI request, and then we have to program the special treatment in these overloaded routines. So, and why use code generation for that? The problems you have when you're doing this
04:02
is that you have an unknown MPI specification, so there are a lot of AD tools out there, CODIPAC is a tool that I am developing, but there's also ALC, GCOC++, ADAPT, FADBAT, and Zaccato, those are operator over tools, every of these tools has a different approach on how to store
04:21
the data, how to evaluate the data, and this has different requirements on the, on the handling you need to do. Tabenade, OpenAD, and ADEC are those transformation tools, they do it a little bit differently, I don't have the time to explain this here, but these have even other requirements in the background you need to handle with your library,
04:43
so if any new guy comes around, you have a new library, I want to do also MPI, you need to change your background interface, and the treatment of MPI concepts is the next problem, you have a lot of different concepts in MPI that reoccur over the full library,
05:01
so if you have implemented some handling in one method, you basically copy and paste it to the other method, and have the same handling there, just make a few tweaks and modifications in order to handle this, so if you do a manual implementation, you have a lot of code copy, which is then actually difficult to maintain, you also cannot generalize this in a library,
05:21
because you have strong interactions between these concepts in MPI, so you always have a little bit of different code, you cannot just tell a function, and also you cannot call these functions in a well-defined manner, you always need something out, and you need something at the end,
05:41
so it's not really doable with a good function call or library definitions, so we need to actually generate some code, so when I started this project, I looked at several different code generators, the one that stuck was GSL, the general scripting language, it's available at GitHub,
06:04
and its input is based on XML files, so you specify some data graph, and then you can use this data graph in order to generate some files, so it's pretty easy to install, it's very easy to switch between the output mode and the scripting mode, where you can compute
06:26
some data, and in this term it's very flexible, so it's not bound to a specific output language, you can generate whatever you want, the cons is there are no IDE integration,
06:41
so you have no auto-completion, you don't know what you actually can to do, and the error output could be improved, let's put it that way, okay, so what workflow I did when I program now the library or what GSL does, so you start GSL with some input, this then reads the XML definition,
07:05
and then basically the best approach is that you first add compute or the pre-computed attributes you need on your XML3, the one you do not need to specify that you can compute out of the ones that you have already specified, afterwards you start to generate the file content so that you
07:24
have still a clear code structure and the template files you write, if you mix this computation of attributes and the code structure you write down, and then you lose the code structure and you don't see anything anymore, so therefore this workflow is
07:44
yeah, this has worked out quite well, otherwise I wouldn't have finished it, so afterwards you really have to apply a file layer, the automatic layouting tool like CLang or some what I'm using, A-style I'm using, so in order to actually have some readable file,
08:05
otherwise you have a different attention everywhere and you don't see anything, and what's really important here, document your expected XML layout, so if you are in a prototyping phase and you have several hundred functions you are wanting to generate and then you
08:24
add one XML property to one function because you need to have something documented immediately, because in this documentation you can look it up and see everything you have to handle and all the corner cases you have to get, if you don't do this you get lost, and also document the generated XML properties because you have no auto completion in your IDE,
08:46
it's not such a large good supported tool and you can look up there what you actually generate and do not always have to go back to the generation code that computes these properties and understands these properties, if you have documented them you know what they are doing,
09:01
so and what's the best practice for progress to implement such code, so the overhead in writing some code that is generated it's like the factor of 10 or 20, so if you can write down some some small function in one minute to write down a code generator that can generate this with
09:26
generalization in there takes about 10 minutes, so first write the example because it's easy, that's what you understand, that's what you can do and have probably been doing for the last couple of years, then test this example if it works, if it works then start the integration
09:43
into the generation engine because then you know it works, you understand the problem you want to generate and then it's more easy to generalize the code, test afterwards and immediately do the documentation afterwards because otherwise you might get lost in the process, okay and repeat this process until you're finished and
10:05
then at the end do something funny in order to reward yourself for the good work you have done, okay good, I want to stress this once more, do the documentation immediately, every change you
10:23
did in the XML structure, some new properties you required and also the new generated properties you have added to your generation file, so now an example how does such a file look, so as you can see here the first mode you have is the output mode in gsl, every line is basically
10:42
put out to the file except the lines that start with the dot, so these are then commands for the scripting language to actually change something or to compute something, so we output here some include iostream, then we change the mode with template zero, you enter the template as you go in the scripting mode and in the scripting mode
11:07
everything is evaluated in the scripting language and if you want to output then something you have to put a right angle bracket here, so then you can go back into the normal mode, have some other commands running here, okay thank you, then you have a for loop over the
11:28
functions that you have defined in your XML file and then you can output here something, so the nice thing is you have always this dollar bracket operator which in every line you output and in every string you have in this language you can add something from your
11:45
XML or add some execution in this string evaluation which is quite nice to concatenate some things, okay so the XML file we have is now here, the root element is functions, this has some function in there with some arguments that we have specified,
12:05
is then executes the gsl command and then we have here our output and as you can see the structure here looked quite okay from the indentation but here you see it's already weird, so have an auto styling tool run over it in order to fix this,
12:26
okay so here a little more elaborated example so you can add here to your arguments some extra properties, can also add some additional arguments to your functions
12:41
and concatenate these and then you can use this to output basically some the overloads that we needed to generate or that the first step in the overloads that we want to generate for our library, okay so here's basically the documentation how I have done it that I have an
13:00
element that's the root element that what other elements it can contain and then you have another element it's a child of the root element what elements can it contain and what attributes can it have if they are optional here I have currently no generated attributes in this thing, so that's how I documented it's just an example to see and this really helps to look up what
13:27
you are expecting in your XML and what you have done or what you're doing in the generator, so now a short look at medipack you can get it at github or at our the side of our here
13:44
of our group it now provides 80 percent coverage of the MPI standard, the missing functions are one-side communication, the init method, the star w methods for functions and so on and this is not because we cannot do it because I don't have any time to
14:06
do it so if you want to use it want to have such functions this functions in there contact me I can do this but currently with this 80 percent we basically cover 99.9 percent of what's used of
14:21
MPI so this is pretty good, so some file statistics the template files have 1300 lines the header files have 5500 lines and we have 13000 generated lines and that's the generator to template ratio of basically 10 percent which is quite okay so the conclusion is the generation
14:41
works also the maintainability works I had to rework it already two times to add new functionality and to fix some bugs and the generation is a really good concept if you have unknown API requirements or if they change over time and if you have several code blocks that reoccur in your code base so thank you very much for your attention
15:08
thank you very much for the nice talk are there questions please
15:30
so so the question is if you have considered source to source translators um that's basically what I'm doing here yes but um because we need some very special
15:46
functionality it's easier to do that way I really have to admit I haven't um um considered this um really big I just said I want to do the library implementation and do the generation but I think that so code to code generation is not really applicable here
16:04
because this has also to work for c++ code and so on and there it's it's hard further questions otherwise may I ask um I think I think you said that the there is no good support by an editor
16:21
there's no code completion and so on right is this an issue that you would like to dress or know is is anybody working on this or any way to make this better because I guess this is kind of hampering issue in this in this case if you want to write a lot of code so I basically um had a vim style sheet adapted to my needs so that I have a nice coloring the completion would be
16:43
the next step yes it would be really some something to do and I maybe want to do it or switch to a different code generator that supports this kind so if you have it done once it's not so easy to switch there