Buildtest: HPC Software Stack Testing Framework
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Serientitel | ||
| Anzahl der Teile | 490 | |
| Autor | ||
| Lizenz | CC-Namensnennung 2.0 Belgien: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen. | |
| Identifikatoren | 10.5446/47240 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
|
FOSDEM 2020365 / 490
4
7
9
10
14
15
16
25
26
29
31
33
34
35
37
40
41
42
43
45
46
47
50
51
52
53
54
58
60
64
65
66
67
70
71
72
74
75
76
77
78
82
83
84
86
89
90
93
94
95
96
98
100
101
105
106
109
110
116
118
123
124
130
135
137
141
142
144
146
151
154
157
159
164
166
167
169
172
174
178
182
184
185
186
187
189
190
191
192
193
194
195
200
202
203
204
205
206
207
208
211
212
214
218
222
225
228
230
232
233
235
236
240
242
244
249
250
251
253
254
258
261
262
266
267
268
271
273
274
275
278
280
281
282
283
284
285
286
288
289
290
291
293
295
296
297
298
301
302
303
305
306
307
310
311
315
317
318
319
328
333
350
353
354
356
359
360
361
370
372
373
374
375
379
380
381
383
385
386
387
388
391
393
394
395
397
398
399
401
409
410
411
414
420
421
422
423
424
425
427
429
430
434
438
439
444
449
450
454
457
458
459
460
461
464
465
466
468
469
470
471
472
480
484
486
487
489
490
00:00
Framework <Informatik>SupercomputerKeller <Informatik>SoftwareSignifikanztestDiskrete-Elemente-MethodeWeb-DesignerKeller <Informatik>SoftwareFramework <Informatik>Vollständiger VerbandSignifikanztestComputeranimation
00:24
Framework <Informatik>Konvexe HülleKeller <Informatik>SoftwareSignifikanztestSupercomputerPhysikalisches SystemBimodulAuszeichnungsspracheProzessautomationNabel <Mathematik>Framework <Informatik>DokumentenserverAuszeichnungsspracheStatistischer TestGemeinsamer SpeicherBimodulPhysikalisches SystemRechenzentrumSignifikanztestKartesische KoordinatenSoftwarePunktSupercomputerSkriptspracheMigration <Informatik>BitWeb SiteComputeranimation
02:00
Framework <Informatik>DokumentenserverOpen SourceElektronische PublikationKonfigurationsraumSkriptspracheBeschreibungskomplexitätMobiles EndgerätOperations ResearchGebäude <Mathematik>Statistischer TestSignifikanztestNichtlinearer OperatorRechenschieberAuszeichnungsspracheKonfigurationsraumSkriptspracheWeb SiteFramework <Informatik>Programm/QuellcodeXML
02:43
Data MiningKonvexe HülleFramework <Informatik>Arithmetisches MittelBimodulElektronische PublikationSchlüsselverwaltungKonfigurationsraumRechter WinkelGebäude <Mathematik>ProgrammMAPNabel <Mathematik>SignifikanztestCodeStatistischer TestFlächeninhaltCompilerCASE <Informatik>Sampler <Musikinstrument>PermutationGeradeVerzeichnisdienstFramework <Informatik>SkriptsprachePhysikalisches SystemComputeranimation
04:00
Übersetzer <Informatik>SkriptspracheKonfigurationsraumQuellcodeFramework <Informatik>Konvexe HülleSignifikanztestGebäude <Mathematik>MaßerweiterungElektronische PublikationVerzeichnisdienstStatistischer TestDatenstrukturKonfigurationsraumDokumentenserverSuite <Programmpaket>MaßerweiterungQuellcodeOpen SourceGrenzschichtablösungOrdnung <Mathematik>
05:05
Konvexe HülleFramework <Informatik>KonfigurationsraumSignifikanztestDeskriptive StatistikCompilerParametersystemCodeFahne <Mathematik>ProgrammierumgebungNabel <Mathematik>KonfigurationsraumOrdnung <Mathematik>SchlüsselverwaltungEinfache GenauigkeitSchedulingGarbentheorieProgrammQuellcodeOpen SourceStatistischer TestMessage-PassingGeradeComputeranimation
06:40
IntelStatistischer TestFahne <Mathematik>BimodulQuellcodeInhalt <Mathematik>Computeranimation
07:34
LastSignifikanztestBimodulFramework <Informatik>LoginFramework <Informatik>Elektronische PublikationSoftwareKeller <Informatik>LastSpider <Programm>SignifikanztestBimodulProzessautomationStatistischer TestGebäude <Mathematik>Computeranimation
08:33
Framework <Informatik>SupercomputerKonfigurationsraumLineare RegressionSignifikanztestSoftwareVersionsverwaltungStatistischer TestDifferenteSignifikanztestLineare RegressionVersionsverwaltungComputeranimation
08:52
Motion CapturingFramework <Informatik>Konvexe HülleExplosion <Stochastik>VerkehrsinformationComputeranimation
09:12
Desintegration <Mathematik>GruppenoperationCodeTeilbarkeitComputersicherheitURLFramework <Informatik>AdressraumStrom <Mathematik>Lineare RegressionSignifikanztestVariableSampler <Musikinstrument>PermutationFahne <Mathematik>ProgrammierumgebungSpannweite <Stochastik>Open SourceÜbersetzer <Informatik>ThreadCodierungIntegralFramework <Informatik>DifferenteGrenzschichtablösungVerschlingungStatistischer TestComputerspielMultiplikationSpannweite <Stochastik>Sampler <Musikinstrument>CodeRoboterEinsInverser LimesSoftwarewartungOffene MengeOpen SourceProgrammierumgebungVariableProjektive EbeneComputeranimation
10:48
Hill-DifferentialgleichungBimodulGebäude <Mathematik>TermStatistischer TestMultiplikationsoperatorLast
12:20
PunktwolkeFacebookOpen SourceComputeranimation
Transkript: Englisch(automatisch erzeugt)
00:06
All right, let's continue with our second lightning talk. We have Shazeh presenting us BuildTest. All right, thank you for joining. My name is Shazeh Bsiddiqui. I am the lead developer of BuildTest, which is an HPC software stack testing framework.
00:23
So let's get started. I'll give you a little bit of background about how BuildTest started. It started around 2017, where I was testing some applications for our HPC system, and we were doing a data center migration. So I was tasked to do a lot of, just basic testing of software.
00:42
So I used to write shell scripts. So I found that at some point, that we need to have a framework that can automate the testing procedure, because I want to be able to share these tests. And the framework has to be tightly coupled with a module system, because we use modules, and a lot of other HPC systems use modules too. I find that writing shell scripts,
01:02
you know, they were hard-coded. I can't share with another site that we also have. We have multiple sites, and I find that if I were to share them with the community, it has to be in some kind of markup language, markup language that can be put on GitHub, and then somebody else can use it
01:21
to write their test on their site. The framework has to be able to automate the test creation and execution, right? So you have several shell scripts. You just want to be able to automate this through some kind of automation tool. And the framework has to be able to provide a repository. A framework without tests is useless, right?
01:42
So have a collection of tests that is community-driven. Now, one point to make out is that build tests are not meant to replace tools like Make, CMake, AutoConf, their purpose are like build frameworks, and build tests is targeting for just HPC software.
02:01
So what is build tests? It is a framework that automates test script creation. It uses YAML as its markup language so users can write their test configuration. The tests are portable. You can use somebody else's test configuration and then use it at your site with adjustments so that it works, and it provides many module operations.
02:23
In this talk, I'll give you two of these operations. And it comes with a repository of tests. On the left, that's the documentation on build tests, so on Read the Docs, and if you're interested in the same talk, extended one, I presented one, Easy Build User Meeting, and you can find the slides in the documentation.
02:44
So I'll talk about kind of like the build pipeline. As I mentioned, so a user right here will write a test configuration. It's in YAML. You'll pass it into the framework. The framework will load the YAML file. The first thing it will do is it'll kind of do a schema check on the YAML file.
03:02
Check the key value pair. It'll check and also parse the YAML file so that it can be set up for the actual build. In the build, it will actually do a staging area where it will pre-create some of the directories where it will write the test, detect the programming language, which is important because it needs to be able
03:20
to find how you can compile the code. Detecting the programming language, then the compiler and MPI if needed, and then it'll generate the compilation line and actually generate the test script. Test script in this case is a shell script. And since we're using modules, build test has several ways you can actually inject modules.
03:40
One of them is active modules and user collections, an Lmod user collection if you're familiar with that. Build test implements its own module collection system and add this to the Lmod user collection, and it has also module permutation as another means. And then finally, it will write the test, and then you can actually run it.
04:01
So how you build the test? Well, the command is called build test build, and then you have many options, and one of them is dash C is for config, and then you pass in the configuration. On the very left, you can see like a directory layout of the test configuration. So when you clone build test, you actually get a repository.
04:21
At the root, there is a directory toolkit suite where I put all the tests, and then you would see a directory structure with subdirectories. One of them, like for instance, you have a YAML file in the tutorial compilers, and then there is a file called args.c.yaml. So that's one of the configuration file,
04:43
and in order to generate it, I mean to actually use it, you're just separating that file separator with a dot so then you can actually run it. You will see there's a source directory, SRC, where you actually have the source code used to actually compile whatever you're compiling, and all the test configuration are in dot YAML extension.
05:05
So here's an example of a test configuration written in YAML. The first line says, okay, we're gonna do a single source compilation, so one source file, and then you compile it. This is gonna be basically a C program in GNU.
05:22
The description key just tells just a brief description of the test, and you can write anything you want in it. The scheduler just tells you wanna run this locally. Scheduler is local, but you can also tie into LSF or SLURM, and then the program section
05:42
is where you actually declare the start of the configuration. Compiler is just a keyword that says, okay, what compiler do I wanna use? And it's just the name of the compiler. In this case, it's GNU. We have to pass in a source file in order to compile. So in this case, it's args.c.
06:00
We can declare the environment variable, so in your shell script, if you wanna say, I wanna say foo equals bar, X equals one. It's just a key value pair. The prebuild and postbuild are sections where you can put in shell commands, so before and after the build, passing in flags to the C compiler, the C flags,
06:22
and then prerun and postrun are shell commands before and after the execution of the code. You can pass in arguments to the executable. In this case, I'm passing in hello world into the program, and that's pretty much it.
06:41
Here's an example of, Intel example of hello world, a different one. You can see there's, you know, you have compiler. You can set it to Intel in this example, and I'm also showing you how you can pass in, like for instance, Fortran flags, in this case, F flags. You can see some of these arrows. What it's doing is actually writing FC for,
07:03
in this case, I4. It automatically finds that out. You have the source file. On the top, you can see how we inject modules. In this example, we're using Lmod user collection, so in this one, a user has an Intel module collection, and then you can see that it's doing a module restore.
07:23
Every test will purge and then load the modules in some way. This example is doing a dry run, so it's just printing out the content without actually writing the test, and if you remove it, it will actually generate it. One of the cool features that I like about this framework
07:41
is it can automate module load testing over an entire software stack. In HPC, you have thousands of software. Well, there's one command I can do it that will test everything. It's called build test module load test, and this example is actually doing it in a login shell, and you can control how many tests you wanna do.
08:02
In this one, I say I'm gonna do five tests. It will just run every single one. It will show you the actual command that's being run, so you can actually copy and run it, and it will tell you which module file is actually being tested. Now, this is actually being done because it's parsing the spider from Lmod
08:23
and being able to find all the modules and then properly finding a way to load them, and if you're interested, I have more examples in the documentation. Build test also has some regression testing. We use a Travis.
08:40
We also test for Python 3.6, 3.7, 3.8, and we have to also test for different Lmod versions, so I'm picking one version of Lmod 6 and 7. We also have coverage report. On the left, you see this is coveralls. It provides a lot of coverage statistics, very in-depth.
09:01
On the middle, there's CodeCov, and then on the far left is the CodeCov bot, so it provides the coverage report in pull requests. And I have to also mention some of the other integrations that we have. So some of the other ones,
09:20
like for instance, we have CodeFactor. For security, we have SNX and Guardrails. We have several bot integrations. If you're interested, we have them in the marketplace. IssueLabelBot and Stale, they're for managing issues in GitHub. Trafico and pull request size are for managing pull requests in your project,
09:41
so how to actually do a workflow around pull request merge, commit, et cetera. The ones that I really like are blackcodeformatter. It auto-formats your Python code, so if you have a lot of maintainers writing codes in different ways, black will just do it in one way, and it really helps your life. And URLChecker is pretty cool.
10:01
You have a lot of URLs, links in your documentation on project. It will just check everything and tell you which ones you need to fix. And some future work. The schema has some limitation. I wanted to be able to do some interesting stuff that it doesn't do right now, so the challenge is trying to make the framework as easily usable.
10:23
So for instance, declaring variables, doing permutations, multiple compilations, multiple compilation flags, multiple runs, permuting them with environment variables or even with multiple compilers. Or if you have one test and you want to run it with a range of values, for instance, like compile some open MP code
10:41
and you want to test it with multiple ranges, or doing multiple source compilation. And I think this is my last slide, and then if you're interested for references, you can check them out. Thank you. Thank you very much.
11:01
So we have time for one or two questions. Any questions? Maybe Kenneth has a question.
11:22
I can come up with a question, yeah. How does this compare to other test tools that are in HPC, like reframe, for example? Do you think it's similar, or? It is very similar. In terms of the building part, yes, it is very similar. I think one thing that distinguishes build test
11:43
from reframe is the way it does the module load problem. So it uses Spyder to actually detect it. And the nice thing is that Spyder will fix all the problems for you with different module naming schemes,
12:00
or even if you install new modules, Spyder's up to date, and that's how it figures out all the modules. There was one question, Todd, I'm sorry. Yeah, we have to switch, yeah. Unfortunately, the answer was too long, so. All right, thank you. Next talk is coming up in three minutes.