Using GNU Guix Containers with FHS (Filesystem Hierarchy Standard) Support
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 542 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/61489 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 20239 / 542
2
5
10
14
15
16
22
24
27
29
31
36
43
48
56
63
74
78
83
87
89
95
96
99
104
106
107
117
119
121
122
125
126
128
130
132
134
135
136
141
143
146
148
152
155
157
159
161
165
166
168
170
173
176
180
181
185
191
194
196
197
198
199
206
207
209
210
211
212
216
219
220
227
228
229
231
232
233
236
250
252
256
258
260
263
264
267
271
273
275
276
278
282
286
292
293
298
299
300
302
312
316
321
322
324
339
341
342
343
344
351
352
354
355
356
357
359
369
370
372
373
376
378
379
380
382
383
387
390
394
395
401
405
406
410
411
413
415
416
421
426
430
437
438
440
441
443
444
445
446
448
449
450
451
458
464
468
472
475
476
479
481
493
494
498
499
502
509
513
516
517
520
522
524
525
531
534
535
537
538
541
00:00
RAIDAreaHierarchyMultiplication signRandomizationTerm (mathematics)Time zoneBitFile systemStandard deviationWhiteboardPhysical systemExpert system
01:27
Computer iconSoftware testingElectronic meeting systemWeb browserOptical disc driveSet (mathematics)Software testingFirmwareInstance (computer science)Service (economics)Numbering schemeIntegrated development environmentModal logicOrder (biology)Different (Kate Ryan album)Physical systemQuicksortPatch (Unix)Function (mathematics)Web browserSoftwareComputer configurationFormal languageoutputDistribution (mathematics)Computer fileOnline helpDefault (computer science)Virtual machineArithmetic meanBinary codeTrailComputer fontStandard deviationFreewareType theoryLibrary (computing)Binary fileGastropod shellElectronic visual displayAuthorizationServer (computing)Scripting languageFerry CorstenForm (programming)Computer programmingBefehlsprozessorMultiplication signBuildingRevision controlCore dumpBootingCache (computing)NamespaceConfiguration spaceMoving averageFile systemFlow separationLink (knot theory)Profil (magazine)Installation artFigurate numberSpeech synthesisSpacetimeBitRight angleBranch (computer science)WordTransformation (genetics)CASE <Informatik>Tablet computerComputer chessLatent heatResultantOperating systemData managementRegular graphElectronic mailing listException handlingNormal (geometry)Information privacyLetterpress printingVariable (mathematics)Computer animation
10:50
Mobile appComputer-generated imageryGamma functionWeb browserGoodness of fitoutputScripting languageLibrary (computing)1 (number)Ocean currentDirectory serviceFigurate numberGastropod shellPrice indexCASE <Informatik>Patch (Unix)Integrated development environmentComputer fileComputer configurationProjective planeSet (mathematics)Binary codeDirection (geometry)Different (Kate Ryan album)Formal languageDefault (computer science)ChainWeb crawlerKernel (computing)Medical imagingBitInstance (computer science)State of matterBuildingWebsiteCondition numberPoint (geometry)Computer animation
14:09
Game theoryGamma functionEuclidean vectorConfiguration spaceoutputMenu (computing)Directory serviceEmailBitSoftware developerGUI widgetoutputCASE <Informatik>Projective planeDirection (geometry)Medical imagingInstance (computer science)Hydraulic jumpRight angleCartesian coordinate systemMessage passingSinc functionIntegrated development environmentSource codeSoftwareMobile appTelecommunicationLibrary (computing)Computer configurationBinary codePhysical systemRevision controlCodeText editorRepository (publishing)Computer fileDirectory serviceComputer hardwareTerm (mathematics)Different (Kate Ryan album)Web browserGastropod shellSet (mathematics)Standard deviationFreewareProfil (magazine)BuildingMultiplication signGoogolCompilation albumExpected valueGame theoryProcess (computing)Scripting languageNP-hardChainCircleNumberSource code
18:54
Gamma functionObservational studyGastropod shellExpressionPatch (Unix)Computer configurationBookmark (World Wide Web)Medical imagingCurvatureSoftware developerAdditionScripting languageFeedbackMiniDiscPhysical systemCASE <Informatik>Graphical user interfaceSoftwareMobile appNamespaceoutputQuicksortMultiplication signCuboidFile archiverPartial derivativeWeb browserSource codeBitLibrary (computing)Virtual machineSystem callError messageOnline helpSoftware testingUtility softwareReading (process)Product (business)Goodness of fitPrisoner's dilemmaWrapper (data mining)Closed setComputer fileProjective planeUniform resource locatorBuildingInstance (computer science)SmoothingElectronic mailing listRootFile systemOrder (biology)Computer animation
24:38
Program flowchart
Transcript: English(auto-generated)
00:05
Hi, everyone. Welcome to my talk about using GNU Geeks containers with FHS support, file system hierarchy support. So it's a pleasure to be presenting here at FOSDEM. Wish I could be there in person.
00:21
This talk, if you're watching it kind of live, is also at a really early time for me in my time zone. So I'll do my best to be mentally present for questions, but I look forward to discussing afterwards. I am not a container expert. I've definitely suffered a bit through some containers and in just trying to make some things work and trying to
00:43
explore other stuff. Containers have been pretty much everywhere, it seems like. So I've come in contact that not not something I've developed personally, but kind of as practical trying to get through and use it. In terms of what FHS is, that stands for the file system hierarchy standard.
01:01
And this is what we typically see in most distros, Linux distros, things in slash lib, slash bin, lots of random things in slash etc, and so on. So that's the typical thing, but this is a rather big assumption. And that's something I didn't even know the term or know what it was
01:22
referring to until I really started working on this and coming to Geeks, especially in seeing what they do there. Also, let me start by giving you a brief overview of GNU Geeks. I'm sure most of you are pretty familiar with it, but those of you who aren't, I'll just give you a quick overview of some of the features and kind of how it works. So it's a distribution of GNU operating system. It follows the FSDG, the free system distribution
01:43
guidelines, meaning that they only deal with free software, essentially. No binary blobs and firmware and things like that in the kernel, for instance. The whole distribution from the package definitions down to the service manager, Shepard, is all built on Guile scheme.
02:04
So, as I mentioned before, I love all things Lisp, and this was really a big feature for me, it's being able to hack on the whole distribution from top down in a language that is a Lisp. And this has brought lots of cool features. Things are transactional, either happens or it doesn't, you don't get stuck in weird states,
02:24
you can roll back to previous sets of packages. The whole system is declarative. So you can just have one file and you declare exactly how you set up your operating system from where file systems are mounted to your users, to the packages that are included, how you configure all sorts of things.
02:42
Including other cool things like transformation, so being able to take a package definition and easily change it to a different branch, a different git commit, to using patches that you have locally to do things, and it's all then way that you can reproduce the same output again, and that's a big feature of what Geeks tries to do.
03:03
And to do a lot of these things, it's necessary that Geeks does not follow FHS. So in other words, in order to have different package versions for different users on the system, they can't all be in slash bin, right? To have different dependency versions or to substitute trying out a newer
03:22
dependency for something you're building versus what other things are, you need to be able to kind of separate out where things are without just throwing them all in one place. This also lets you do the system configuration and kind of roll back and change things just by changing symlinks, basically. And I'll just leave a few links up here for those who want to read more.
03:45
So, speaking of, then something that's a nice feature of Geeks, which is what this talk is about, kind of an additional option, is Geek Shell. And this is essentially a very quick one-off environment. You can just install something in this temporary environment, this shell, and use it, do some testing, do whatever you want, without
04:05
installing it in your main profile. So rather than normally, in most issues, you want to use something, you have to install it, use it, and then you forget about it, right? I've definitely installed lots of little tools just to play around or try something, figure out which one I want, and then forget until you realize you've installed lots of stuff taking up space.
04:23
So this lets you do a kind of a one-off thing. Some nice features too is that this is cached, so the first time it may have to download subsuits or build things if you're building locally for whatever you need for the package you want to run. So that can take a little bit of time, but afterwards then it's nearly instantaneous, right?
04:42
If this is something that Geek Shell's already computed, the set of packages you want, it'll run pretty much as quick as just running the stuff directly, the launching at least. After that, it's, you know, however fast the program runs. And I found this really an invaluable tool. I love having little tools I use once in a while. I don't need them around all the time.
05:02
So, some simple examples here. One is for using, let's say, Python or another language where you just have some scripts you want to run occasionally. You don't need to develop them all the time, you don't need Python and tons of Python packages all around. So in this case, you can do Geek Shell with Python as an input and Python,
05:23
here's a particular package for that, Canvas API I use for some grading stuff. And then after the double dash, the command you want to run in that shell. You can also just do Geek Shell without the stuff after the two dashes to enter an environment there for as long as you need and then exit back out to your normal shell.
05:42
So, nice simple tool. It's really powerful. It's great for building up one-off environments or being able to reproduce a set of packages to hack on something, for instance, without having to maintain all of that at once. So, the next step on this command is the container option. So, the long forms dash dash container dash C, and it runs it, as you might guess, from the good naming in a container.
06:07
And this container uses pretty much the same technology as everything else, which is namespaces. It works basically the same way where you are in an isolated environment, so you have to specify everything you want to be in there.
06:20
It's not quite a virtual machine because we're not emulating, for instance, a different host CPU or something like that. But it's kind of in between there and you specify everything you want. It's by default in a container, so it's isolated from the host. It lets you do things in a reproducible manner and a way where you can keep things contained and not have to worry about being polluted by environment variables, other packages, anything else you have going on.
06:48
So, the new option that I'm going to spend the rest of the time on this talk about is FHS containers. So, this is a new option building off of that container option called emulate-fhs or dash-f for the short version.
07:01
And in short, this makes things in the container look like an FHS distribution. So, we'll get things like slash lib, slash bin, and it also includes on a more technical note a glibc, which will read from a global loader cache. That'll help with compatibility, which is one reason why it was added here.
07:20
And in short, the uses here is it gives you a nice minimal environment that's more typical in a sense. Unless I love Geeks and think it can take over the world, it hasn't yet. So, it's good to be able to make contact with what most other people will be using in some way. Often language-specific tooling expects to be able to download packages from places or to
07:42
set up an environment in a certain way which may not play well with Geeks. So, if you're kind of in between using packages in Geeks or having to use other ecosystems, that can be a handy way to set up a nice isolate environment for that. Likewise, for some binaries, I'll discuss testing too. If you want to be able to test something in a different environment, not going to
08:03
a full virtual machine or anything like that, but sort of like a chroot type testing setup. So, before I get into some more specific details. If we look at our previous one, we add here the FHS option. Here we can see that slash lib has a whole bunch of stuff.
08:24
Bin also has a bunch of things, as we might expect in a typical place versus here I see nothing. And in a regular Geeks system, I just have a symlink for shell for a compatibility reasons, right?
08:40
That's it. Everything else needs to kind of come about through those profiles, the symlinks I mentioned before, to be able to find things. So, the FHS environment, in short, just kind of sets up some further symlinks and puts things that you'd expect to be in certain places where things can find them. So, let's look at a few examples.
09:00
I think it's the best way to demonstrate the uses and kind of how this is a nice, neat tool to do some things. So, one, the Tor browser. This is one where you're usually concerned about privacy if you're using it, fingerprinting, and in other words, tracking down who someone is based on things like font sizes, things that are installed, canvas sizes, et cetera.
09:20
So, for privacy, Tor, for instance, and other places will recommend running kind of a standard browser, right, without all these little things that users do to make it their own. So, in this case, running the official Tor browser binary is a good idea. And why not? We're trying to be safe and private. Let's have some extra isolation and keep that environment pretty self-contained.
09:44
So, here is the command here. In this case, I've downloaded the Tor browser. I've already extracted it to a folder called Tor-browser. And then this command just to highlight some of the different pieces there. So, running geek shell in a container, the network option gives network access.
10:02
By default, we do not do that, and the FHS option. Now, in order to do things like display something on the host outside the container, we want to provide some environment for it. So, preserving and sharing things like the display and X authority for an X server will let us do that.
10:23
And then here, Tor browser is pretty self-contained, but it still expects to have things from the typical distro that's being run on. So, here I've added ALSA, bash, core utilities, various things there, as you might expect, to get things to run. And then we can just launch it, basically, which is this last piece.
10:44
So, let's try that out. In this case, it starts up right away. I've already run it, so it didn't have to download and set up all those packages. And there we are. We're in our Tor browser. It's very bright, but looks good.
11:02
So, another example I mentioned earlier is kind of tooling ecosystems from different languages. Rust is one which is very popular. It's in the Linux kernel, or just about to be. And it moves quickly, and a lot of projects you'll notice will use the nightly set of toolchain binaries and libraries and all that stuff.
11:21
So, if you're keeping up that, developing things, you may want to have access to that. In Geeks, we're not as quick on updating Rust. There's a lot of things that need to be built, and it's kind of a moving target. So, in this case, maybe you want to do that in a separate environment, be able to use the kind of quote-unquote usual tools and directions that someone gives for setting up Rust.
11:44
So, in this case, there's the RustUp tool, which is just a little script that will download and set up a Rust toolchain and environment for you. As you might expect, if you tried to do this, I suppose I can. Why not? This is the instructions given on the RustUp website, just says to kind of curl and pipe that into shell.
12:05
So, if we were to try that, it'll download it, and then you'll get a cryptic no-such file or directory error, which is weird because if you look, the thing it's asking for does exist. But in this case, it's indicative of it's trying to run something or use a library that's not where expected to load it, in this case.
12:26
So, there's ways around this. You can patch the paths to point to the right place and so on, but that's getting a bit tricky. Instead, we'll use our new tool. So, here I'll run a shell, again giving network access to download stuff,
12:42
and then a bunch of inputs needed. In this case, this is a lot more than you need just for the RustUp script. For the RustUp script, you pretty much just need, I think probably GCC lib to load stuff, the toolchain for building stuff, curl, grep. The rest of these more graphical ones are for building, as an example, a Rust project, which we can show as well.
13:07
So, here, run this. And the last option it mentioned here, it says I'm sharing the temp home directory I created as home in the container. So, by default, the container will just see the current working directory.
13:21
So, in this case, I eliminated the current working directory so it doesn't kind of appear nested. But that's kind of the default behavior. You'll just see the directory you're in, not anything else outside of it. So, for instance, if you want to reuse your environments, especially an FHS one where you might be running things for a while and want to go back to it, then you'd want to set up a home directory for it.
13:43
You could share your own. I think it's good practice here in containers, set up a separate one and let that build up the state for that. And then if you want to erase it or go to a different one, you can just change that option and not have to clean out things and figure out what got touched. Okay, so then let's run this.
14:03
You can see it has some instructions it already says. And we can just let it download things and run. That's it. It gives you some instructions here to source something. And we can see if there's a Rust. And indeed there is. 1.67 from five days ago. Pretty recent.
14:22
Outside the container, I don't have Rust. But then from there I can follow the usual directions for a project. So the inputs I gave here was from an example of EWW, this widget library. It's pretty popular for making kind of cool desktop widgets.
14:41
And in that case it uses the latest Rust. And the instructions for building the project are pretty much to clone the repository and then run cargo build. And you can do that in the shell once you have Rust up that gave you the latest version and sets up your environment the way you want it. Without polluting and messing up your main environment and shell.
15:01
Which is quite nice, especially things like this which are downloading a lot of stuff, setting things up. And if you especially want to test things, build from a clean environment, this is just a really nice tool for doing that. All right. As another example addressing something in Geeks in particular, but I think pretty handy more generally, we don't have Electron-based applications.
15:23
Really the JavaScript packaging nightmare, dystopia as I've heard it called, is just hard or impossible right now to package things from source or to bootstrap it from source all the way through. Just the dependency chains, circular dependencies and the ecosystem there is not really built with what
15:42
Geeks wants to do and the standards we have for how we want to package things. So you have some free software Electron-based stuff for instance which there's no reason why you shouldn't be able to run it but we can't build it from source. But we could use app images for instance which are supposed to be these nice self-contained, have everything in them packages.
16:01
As we'll see, that's not quite as simple as that. But let's jump right to an example before looking at some detail there. So in this case, I have a little bit more, a trick that I want to show here is using the development option. So what this does is it grabs all the development inputs needed to build in this case on Google to Chromium from source.
16:24
So in that case, it will be all the libraries, the compilers, all the stuff you need if you were to be working on that project. In this case, I want to grab all those inputs. And this I just found is a nice kind of, not a finessing tool, not a minimalistic tool. But as a way of grabbing lots of inputs when you don't want to mess around with things
16:42
or you expect that you'll need all the kind of stuff that a typical browser does in this case. So for Electron things, I think that's helpful. GCC lib is usually needed for a few libraries that are always expected to be around for binaries for instance. And now we start getting to some more details for kind of desktop applications.
17:01
They'll often expect to have access to D-Bus and be able to send messages and receive messages that way. So preserving that environment and exposing where it runs. Likewise, in this case, since it's using Chromium basically as a rendering engine, it wants, tries to use hardware acceleration. So exposing a bunch of devices and other hardware things is needed to make things work as smooth as possible.
17:29
While most of this is, I would say, kind of reproducible in a sense, some of these options are getting probably more particular to my system in terms of what hardware is needed, what it tries to run with.
17:41
So some of this may need tweaking on different systems. But once we have all that, then I've downloaded from some weeks ago a version of VS Codeium, which is the freely licensed, free build of Microsoft's VS Code editor. And once you've made it executable, you can just, it's supposed to be able to just run it directly because it's supposed to have everything in there.
18:07
But that doesn't work, so maybe we should try that first as an example just to see what that looks like. So I get the same no-such file or directory if I try to just run it.
18:21
As expected, binary assumes you have some things. It doesn't expect you to have nothing. So they're not really a self-contained. And I've seen other app images when I've had to include other random inputs you'd expect to have packaged in there. But other times, they make assumptions on what's on the host system. Which is why here I've included some other stuff that, again, is overkill probably for this package, but just as an example that you can use.
18:51
So again, the profile is already existing, so it doesn't have to do anything here. And we get VS Codeium. So a couple things to note here.
19:01
One, as I mentioned, I've exposed more stuff from the host in order to get this to run. So even though this is a container, it's supposed to be self-contained, the app image, you still need things from the host to run. A big graphical tool, desktop tool in this case. So it's something, you know, it's a convenience to be able to run things like this on the host without having to build it from source and all that.
19:25
On the other hand, it's not really completely contained and private if we have to start poking lots of holes in order to get things to work. On a kind of technical note, the app image here was using this option called app-image-extract-and
19:41
-run, which basically the app image is, as far as I understand, sort of an archive slash disk image. And so normally when you try to run it directly, it mounts itself using FUSE and then, you know, that's mounted somewhere in your file system and then it runs from within there. This doesn't work because if you try to mount something within the container, you don't have access.
20:03
Usually you need root access or FUSE access in this case to mount something. You don't have that from within the container. There is a way to call out to the container using tools from Flatpak. In that case, you can, as a test I did, you have a little wrapper that will call FUSE mount on the host to mount the image.
20:27
And that actually will work except that within the container you don't see the mounted image. I would love if someone could explain the details. Basically, I'm thinking of something to do with when namespaces are created and since the container already has access to certain things, even if it has access to where that image is mounted,
20:45
through FUSE, it can't access what's in there. It sees nothing. If you create another container, if you look from the host after running this mount, you will see it. So that's a little bit of a technical detail there on kind of containers, which probably someone can explain a little better.
21:01
But for good reasons, generally you don't want things in the container to call out to the host, especially things that need special access like mounting disk images. Alright, so a few tips I want to kind of close out with. In general, the packages you need, it's not clear. Usually there's a lot of trial and error.
21:20
You run something, it complains it can't find a library. That's not always the most helpful because often you'll get, as you saw, like file not found or you'll get some other error and it's not clear where it's failing to load something. So in that case, strace can be a bit overkill, but you could of course use that or other tools to kind of see what libraries are trying to be loaded and where it's breaking down.
21:42
Readme, surprise surprise, can often tell you what is expected to build a project for it to run, but they're not usually complete. There's usually some assumptions of tools everyone supposedly has on a machine in a distro. So there's a bit of trial and error going on there.
22:01
Xdgutils from Flatpak lets you kind of call out to the host using portals, as they're called, which is a way of then, for instance, passing a URL to be opened on the host browser. Lastly, what's next? I think some utilities to make this easier to script. You can definitely take those long, quick, geek shell commands and put them in your favorite script and run them that way.
22:26
You can use a kind of longer shebang to also call geek shell and so on. But I think it would be helpful to have some ways of kind of packaging up some of these common options or ways to run things like that more seamlessly from the host. So I think that could be some things we could work on and can make things kind of smooth.
22:46
I'm also interested to hear what uses other people have. A few people already on the geeks mailing list and IRC sometimes chime in with things they're trying to do. And that's been helpful to see what works and what doesn't. Yeah, so to end, I think this is another great tool in the geek shell toolbox.
23:04
I know I'm a little partial as having written the patches, but with help. But I think it's just something that lets us do a lot of stuff in geek shell that, for practical reasons, we want to be able to do. I would love to be able to build everything from source, to have it reproducible, to have a geeks package for it. Not always possible. It's not something I really reach for very often.
23:25
Very few occasions I've needed it, but it's great to have it there, be able to test something that I might expect to work on another machine, for instance. But it's always, I think, a good learning experience. This has taught me a lot about kind of geeks about what's reproducible, what's minimalistic, to come back to the theme of this dev room.
23:43
Being able to really specify everything that's in your container and what's needed and understand really what's happening there gives you a good understanding, I think, of how software is built and the shortcomings. Even things like app images and flat packs, which are supposed to be all in one, are not really. And this gives us another way of kind of running stuff like that or being able
24:04
to maybe develop ways of packaging things like that in addition to other tools geeks has. But I definitely appreciate any input and feedback and questions people have. And just to end really quickly with a thank you to Ludovic, especially, who helped really tweak and polish these patches and some fixes.
24:24
Previous work done on a third-party channel, which is non-free, so I won't mention any detail, but that was kind of the origin of some of this stuff and things that I worked on there as well. Thank you everyone for paying attention. It's been a pleasure to be here. I hope to see everyone in person at the next one. Thanks.