Playing with Nix in adverse HPC environments
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 542 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/61611 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2023145 / 542
2
5
10
14
15
16
22
24
27
29
31
36
43
48
56
63
74
78
83
87
89
95
96
99
104
106
107
117
119
121
122
125
126
128
130
132
134
135
136
141
143
146
148
152
155
157
159
161
165
166
168
170
173
176
180
181
185
191
194
196
197
198
199
206
207
209
210
211
212
216
219
220
227
228
229
231
232
233
236
250
252
256
258
260
263
264
267
271
273
275
276
278
282
286
292
293
298
299
300
302
312
316
321
322
324
339
341
342
343
344
351
352
354
355
356
357
359
369
370
372
373
376
378
379
380
382
383
387
390
394
395
401
405
406
410
411
413
415
416
421
426
430
437
438
440
441
443
444
445
446
448
449
450
451
458
464
468
472
475
476
479
481
493
494
498
499
502
509
513
516
517
520
522
524
525
531
534
535
537
538
541
00:00
SupercomputerIntegrated development environmentCompilerHochleistungsrechnenRun time (program lifecycle phase)Task (computing)BenchmarkBefehlsprozessorRun time (program lifecycle phase)Online chatBitFlow separationBefehlsprozessorSystem callCompilation albumComputer virusWorkloadComputer animation
01:34
Database normalizationNormal (geometry)Block (periodic table)State transition systemStack (abstract data type)LoginVertex (graph theory)RootkitVirtual machineNP-hardRevision controlControl flowComputer configurationConfiguration spaceLibrary (computing)IterationFocus (optics)Open sourceVisualization (computer graphics)PlotterDerivation (linguistics)Service (economics)Thermodynamisches SystemWrapper (data mining)SpacetimeScripting languageIntegrated development environmentRun time (program lifecycle phase)Execution unitTelecommunicationProcess (computing)Gastropod shellDemonLattice (order)WorkloadDefault (computer science)Patch (Unix)Latent heatAttribute grammarInheritance (object-oriented programming)CompilerCompilerCompilation albumMechanism designIntelBenchmarkoutputBuildingComputer programmingTelecommunicationFocus (optics)NamespaceGastropod shellLoginState of matterProcess (computing)Link (knot theory)Structural loadWorkloadData storage deviceKernel (computing)CASE <Informatik>RhombusBitRevision controlMultiplication signScripting languageRun time (program lifecycle phase)NumberMathematicsFunction (mathematics)System callLibrary (computing)Configuration spaceCycle (graph theory)Compilation albumFunctional (mathematics)Parameter (computer programming)Set (mathematics)SoftwareLatent heatNetwork topologyPatch (Unix)HookingOptisches KommunikationsnetzThermodynamisches SystemEmailSoftware developerDerivation (linguistics)Computer fileComputer configurationSoftware testingResultantVirtual machineResource allocationPoint (geometry)BootingClient (computing)DemonVisualization (computer graphics)Fiber (mathematics)Computer animation
10:10
Cache (computing)Software developerCompilerConfiguration spaceOpen sourceElectric currentIntegrated development environmentSoftwareThermodynamisches SystemHost Identity ProtocolSoftware testingSpacetimeGastropod shellScripting languagePhase transitionSubsetBefehlsprozessorWide area networkComputing platformLatent heatDemonSupercomputerComputer configurationData storage deviceMultiplication signCartesian coordinate systemComputer programmingModal logicNamespaceFlagBitCompilation albumRootOpen sourceWeb pageThermodynamisches SystemComputing platformMathematical optimizationCASE <Informatik>Perfect groupLibrary (computing)SpacetimeMiniDiscStandard deviationCycle (graph theory)Roundness (object)Wrapper (data mining)Virtual machineBuildingComputer architectureSupercomputerMathematicsSoftwareExterior algebraSlide ruleDemonDynamical systemScripting languageSigmoid functionSpeicherbereinigungModule (mathematics)Computer fileConfiguration spaceProcess (computing)Computer animation
18:45
DemonSupercomputerComputer wormComputer animationProgram flowchart
Transcript: English(auto-generated)
00:14
Hello, and for the next talk. Thank you, hello.
00:24
I hope you can hear me online well, if not complain on the chat. Okay, so we are going to talk about... No? You can really yell, that's fine. Okay. We are going to talk about HPC, high performance computing, and Nix,
00:43
and how we kind of deal with that. My name is Rodrigo, and my coworker Raul is here. First, a bit of what we do. Essentially, we work on a parallel concurrent task-based runtime,
01:00
similar to OpenMP, if you are familiar. We also need to work with a compiler based on LBM to read these pragmas in the code, and transform them to function calls to the runtime. In our job, the performance is critical,
01:21
so we really need to take care. And in general, we execute the workloads on several hundreds or even thousands of CPUs. Here's a little example of something that we have observed. We have a program here that runs,
01:42
and here you can see the CPUs, and this is the time of execution. And we are examining this little point here, because the time here is slightly bigger than what is normal. And we can see that the problem is that the allocator took a bit longer.
02:02
So this is just an example. In general, in H2C or this high-performance computing, it's just a lot of machines connected by a fiber optic. They are managed by this SLUR diamond, which allows you to request a certain number of nodes.
02:20
We don't have fruit in any node. In general, it's very old kernels and very old software stack. So, yeah, we are stuck with that. And in general, the state of the art now is to use LD library path to load other software and change versions. The problem with this technique is it's not very easy to reproduce.
02:45
So the question is, can we benefit from using Nix? In general, we will get up-to-date packages, configuration options. For every package, no more LD library path, and we can track everything that we use for an experiment.
03:02
The problem is we don't have fruit, so we cannot install the Nix diamond, as we would like to do. So let's take a closer look at what we do and how we do it. In general, we work in these three huts, so to say.
03:22
In the development side, we take a program, and we compile it several times until it actually compiles. We kind of need to do this cycle quickly, so we want the compilation time to be very low. So we need to reuse the already built tree to run the build command.
03:43
When we are finished, we switch to the experimentation side, and we run this program in the machine. And maybe we need to tickle with the arguments or the configuration file of the program to get some results that we want to examine. And then we also do some visualization of the results,
04:00
but we are not going to talk in this talk about this. So we will focus first on the experimentation and later in the development side. So a bit of what we did. We tried with individual installation of the Nix store by using user namespaces.
04:21
The problem is that the number of packages grows, so we would like to share the store with several users. So we use an auxiliary machine where we actually have a Nix daemon, and then we can perform the build in that machine, and then use the post-build-hoc to execute some script
04:43
that copies the output derivation to the actual cluster. Problem is, inside the cluster, Nix store doesn't work. So we wrap the command Nix store in a shell script, and when it's invoked by the auxiliary machine,
05:02
it creates namespace where it mounts the Nix store there, and then it runs the Nix store and receives the derivation, so we can actually copy it over SSH. We also tried to patch the Nix daemon to run inside the machine, but it's a bit complicated, because we cannot even run a user daemon there.
05:26
Okay, so let's focus on the experimentation cycle. The first requirement, the most important thing, well, assuming that you already have a program that somehow you built in a sandbox,
05:41
we want to execute this program in the machine, and we want to make sure that this program doesn't load anything that is outside the Nix store. So, especially the LD library path may have some path that actually has libraries for your program, so we don't want that,
06:01
and also it may use the dealopen to load other libraries. So, ideally we want something like the Nix boot with a sandbox that prevents access to slash user or a slash opt. And it needs to work in a SLORM too. Another requirement that we need is for MPI,
06:23
the communication mechanism, to use this syscall process bmreadby that only works if the process are inside the same namespace. So we solve this by running a check that checks if the namespace is already created, and if so, we enter it, otherwise we create another one.
06:45
So let's take an overview of how this works in the cluster. We have here the login node and two compute nodes that were given to us for running our program. In general, we have to wait a bit after requesting the nodes, that is fine.
07:04
After this moment, we take a shell that is connected to one of the allocated nodes. These are the nodes, and each node in your case has two sockets. So we usually run one process per socket, and we talk to one of them only.
07:23
Inside this process, we don't have Nix. So we first load this namespace by using our script, and then we can run other programs like SRAM, which is the client that will launch the workload
07:41
that is inside the Nix store. So we can compile programs, link it to this specific version of Nix, of Slurm, sorry. After that, it requests the Slurm diamond to execute something in parallel, and the Slurm diamond forks in every process,
08:01
one process that will run something, but it's outside the namespace because it's not controlled by us. So we execute our script again to join the namespace if it's found, otherwise we create another one, like in the second compute node. And here we can see that we can communicate
08:24
in the same node, because they are both in the same namespace, and they are one-two, and here we use fiber optic communications. Another requirement that we need is that we need custom packages, and we use that with this technique
08:42
where we define a call package function that takes priority over all attribute set. So we can change software that is provided in upstream with Nix packages, and we use our version first, so we can hack on those
09:00
without disturbing the whole package set. Another thing that we need is to define package with compilers. In general, we use LBM with a custom runtime, so we use the wrap-cc-width and inject this little environment bar so we can load our runtime
09:21
without needing to recompile the compiler. We also need, unfortunately, proprietary compilers, and we use this RPM extract and the auto-patch of hook to fix the headers so we can run them on Nix2 and compile the derivations for them.
09:49
Now we will talk about development cycle. Okay, let's move on to development cycle. In general, the development process consists in getting an application,
10:02
adding some new and cool features to it, breaking things, testing and re-testing that is okay, and this interactive workflow requires frequent changes in the source and compilation steps. For this reason, Nix build is not good to work with
10:20
because every change in the source will trigger a full copy of the source to the Nix store and a full compilation. With big repositories, this is a problem because, for example, in the slide we can see how much time it takes to build LLVM in a 32-core machine we have,
10:41
so it's a big machine, and we can see that, although we use CCADs, we talk about different orders of magnitude, comparing it with simply reusing the previous build. Another alternative could be using Nixel to get our tools to build the application,
11:02
but this environment is not isolated from the system, and we can find software that includes hard-code paths that lead to the system, like in this case with a sigmoid module file of ROC-EM that is good enough for EMD, for those who don't know what it is.
11:23
And if we take an application that uses ROC-EM and configure it and check the log output, we can see that, at the end, the installation selected is the system one, instead of the Nix package we want to. An isolated environment will prevent us from this situation,
11:43
avoiding the necessity of patching the source to solve this problem. Our solution for these two requirements is to first build an isolated environment with a tool we named Nixwrap. Nixwrap is a script that uses bubble wrap
12:02
to enter a user namespace where the Nix store is available, but not the system directories, like in this case, the slash user. And in this environment, we can launch our Nix tools, like, for example, Nix Build. And this works because inside the namespace,
12:22
Nix Build creates a new sandbox in a nested namespace, so the environment is not affected. And the most powerful feature of it is running Nix Shell inside this isolated environment to get your tools to build your application in an isolated environment
12:41
so you don't have to worry about accessing to the system. And in this case is the previous example, LLVM, and reusing the build. And finally, if you are using like SLURM, you can execute your application by running Nixwrap
13:04
after the SLURM step for process and your application. Another requirement for us is since we are in an HPC environment, we want to get the best performance of the applications. And for this reason, we need to build the critical performance software
13:23
with CPU optimization flags. Our solution for this situation is to override the compiler wrapper injected flags by overriding the host platform attribute, specifying the architecture and other stuff to the compiler in the standard environment NoCC.
13:43
And finally, we create the standard environment we will use to build our software with this compiler wrapper. So, I will talk about the conclusions.
14:02
In general, we can actually benefit from using Nix. But obviously, we have some drawbacks. These cycles that I was talking about, we can still do it very fast. So yeah, it's very nice for us. And also, if we have the chance to get something like a Nix demo,
14:20
without the root requirement, and still be able to share the Nix store, that would be awesome. Thank you very much. We have five minutes left for questions. If there are questions.
14:41
When I was working in the HPC environment, there was always an issue with disk space. How does it work with a dynamic Nix store where people could just, say, donate anything into the store? Can you repeat the question first? Yeah, so the question is, how can we manage to Nix store where users can install,
15:02
if that can be an issue for disk space. So, in general, right now, we have about 300 gigabytes of storage. For our particular group, we have around two to three thousand gigabytes of space available.
15:24
In general, in HPC, people used to use a lot of the space. But if we share the store, that will be the best solution instead of every user to have their own installation. And we also, when we kind of analyze someone,
15:43
that says to us, please use less space, we run the garbage collector. Thank you. Yeah. So, you said that the state-of-the-art was people using LD library path on the machines. Did you consider using, or can Nix use R path
16:05
instead of using run path? Because that would get rid of your problems there. That and then the other thing you can do is there's a talk in the HPC webinar about rebinding the path to SO. OK. But that's a little bigger hammer. OK.
16:21
I see your SPAC T-shirt from here. OK. So, question is about using R path. R path, yeah. Because it takes precedence over LD library path. You don't have to worry about the user being stupid. Yeah. So, the problem is that you can see programs
16:42
using dlopen to load their own... They don't... I'm sorry? dlopen respects R path too. Ah. OK. I didn't know that. We're doing something wrong. OK. Unless you can do like a define, what does it matter?
17:07
OK. So, dlopen is not only the only problem because we also see software trying to read its C slash configuration file somewhere. And we also want to prevent that.
17:23
Yeah. In general, we saw that it is safer to avoid the programs from accessing any path than trying to find every single option that the program can use to access. There was still one eager question over there. Can we find the next rapid script online?
17:40
Yeah. I think I will upload it to the Folsom page. Any other questions? Not so much of a question but a bit of a shameless plug. The main blocker for having a rootless Nix daemon was merged last week or the week before. So, hopefully that's going to eventually solve the third of your points.
18:04
Perfect. Thank you. So, what about interior libraries on the system? Are you only envisioning that you would install the library, things like MPI, through Nix? Because that's not a possible one. Yeah, it's a very good question.
18:20
For now, we have been very lucky to be able to work with only proprietary packages that can be put inside Nix. But it may happen that the proprietary something, it doesn't work. So, we don't have a solution for now. One more round of applause. Thank you.
18:40
Can I just switch it over again?