Your Distro is a Scheme Library
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
Title |
| |
Subtitle |
| |
Title of Series | ||
Part Number | 106 | |
Number of Parts | 110 | |
Author | ||
License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
Identifiers | 10.5446/31021 (DOI) | |
Publisher | ||
Release Date | ||
Language |
Content Metadata
Subject Area | |
Genre |
00:00
Library (computing)Module (mathematics)Right angleNumbering schemeData managementType theoryElectronic mailing listSoftwareMereologyLink (knot theory)Latent heatProblemorientierte ProgrammierspracheObservational studyObject (grammar)FrustrationBitQuicksortNormal (geometry)Maxima and minimaFunctional (mathematics)Software developerExtension (kinesiology)Point (geometry)Graph coloringInterface (computing)Distribution (mathematics)Flow separationMultiplication signInstance (computer science)Initial value problemTouchscreenDatabase transactionNetwork topologyComputer programmingProcedural programmingProgrammer (hardware)Web applicationUser interfaceThread (computing)Mixed realitySlide ruleData storage deviceSkewnessWeb pageCoroutineVideo gameCartesian coordinate systemPresentation of a groupActive contour modelSocial classProcess (computing)Field (computer science)Web browserDirectory serviceOpen sourceIterationNumbering schemeTerm (mathematics)Abstract syntax treeCodeInstallation artElectronic program guideFormal languageBuildingSoftware repositoryXMLLecture/Conference
09:38
CodeDerivation (linguistics)RootDemonSystem callGraph (mathematics)Data storage deviceComputer fileElectronic mailing listSource codeUtility softwareConnected spaceBitLevel (video gaming)Numbering schemeSoftware repositoryAbstractionoutputObject (grammar)Flow separationExistential quantificationTheory of relativityModal logicFluxLibrary (computing)Information securityInstallation artType theoryException handlingGreatest elementClient (computing)Integrated development environmentBuildingRemote procedure callCore dumpPoint (geometry)Bootstrap aggregatingBinary codeProcedural programmingGraph (mathematics)NamespaceRight angleCuboidParticle systemCross-correlationExecution unitValidity (statistics)Software developerResultantSlide ruleParameter (computer programming)Order (biology)Endliche ModelltheorieGoodness of fitCoefficient of determinationImage resolutionInstance (computer science)Electronic program guide1 (number)Process (computing)Traffic reportingCompilerPower (physics)ArmConfiguration spaceMaxima and minimaRule of inferenceMusical ensembleScaling (geometry)Field (computer science)
19:12
Link (knot theory)Core dumpParameter (computer programming)Operating systemExpressionComputer programmingDifferent (Kate Ryan album)System callProcess (computing)SimulationDerivation (linguistics)Mechanism designFraction (mathematics)Revision controlPoint (geometry)Multiplication signLevel (video gaming)Configuration spaceCodeDefault (computer science)TouchscreenElectric generatorPhysical systemFunction (mathematics)Complete metric spaceComputer fileDeclarative programmingVirtual machineHash functionWritingService (economics)Data managementDemo (music)Numbering schemeSimilarity (geometry)outputTerm (mathematics)Functional (mathematics)InformationMonad (category theory)Electronic mailing listMereologyLatent heatRight anglePattern languageData structureBuildingArithmetic meanField (computer science)PhysicsPresentation of a groupExecution unit1 (number)Arithmetic progressionResultantTwin primeInstance (computer science)NeuroinformatikOcean currentCASE <Informatik>Roundness (object)Linear regressionProjective planeRule of inferenceData storage deviceIntegrated development environmentForm (programming)Set (mathematics)Computer animationLecture/Conference
28:45
Computer animationLecture/Conference
Transcript: English(auto-generated)
00:05
Let's start, maybe. OK, so thanks, everyone, for coming. It's really nice to see so many people in this room. I really appreciate it. So basically, since this is a guide dev room,
00:21
I thought I would show a little bit of scheme code this time. So you had the opportunity with Ricardo to learn a bit about the user interface of Geeks. What I would like to show you is what makes it different. You can always use Geeks from the command line. That's what we do most of the time. Or from Emacs, which is very important.
00:42
But there's more to it, right? You can use Geeks from Scheme, which is sort of like a command line in the end. And it allows you to do more things than you would usually be able to do with other tools, right? Welcome. Welcome.
01:01
So OK. I think Chris is going to laugh at me with this slide. I'm really sorry. But probably you've seen this phrase before. We're claiming that we're building the Emacs of these tools. And it's not just because we're nerds.
01:22
I hope. OK, I'm a nerd, but still. I mean, it's all about being able to actually hack the thing and understand how it works and be able to fiddle with it and get into the internals. And we really want to blur the distinction between users
01:43
and developers, right? I mean, you can always stay at the command line when you use Geeks. And you can already do quite a lot of stuff. But it's very nice. And I hope not too hard to actually dive into the Scheme interfaces for Geeks.
02:00
Like Ricardo showed us an example of how you can customize a package definition, right? I mean, package definitions themselves are like, you know, it could be XML, right? It's purely declarative. And then you can start modifying it. And you're actually writing Scheme code, hopefully without noticing. And that's the whole idea behind Geeks.
02:21
So it all goes back to what Stalman was attempting to do with Emacs back in the day, which is that, you know, Emacs is programmable. And the idea is that people would start programming without noticing. And that's really what we're trying to do. So yeah, I'm going to show a little bit of that.
02:44
So there was a talk by Marco at Liber Planet last year, I think, where he was mentioning, well, user freedom is really two things. We have access, you know, that you're able to actually have the software and to use it in any way.
03:01
And there's empowerment, which is the third and fourth freedom, that you want to be able to modify the software to do what you want, basically. And this is empowerment. And Marco was saying, we're doing pretty good with access. I mean, everyone is using free software, maybe not everyone. But a lot of people are using free software.
03:21
Unfortunately, sometimes without knowing. But we're not doing so great at empowerment, because it remains hard to modify most pieces of software. And so Geeks, in a way, is a humble take on this part. We're trying to make it easier to get into the software
03:42
and actually modify it. So you can have practical freedom. You can actually know what's going on and modify it. So I think to get some sort of a guided tour of what's inside Geeks in terms of APIs, we
04:00
could start by looking at this command and ask ourselves, what's going on when we type this? So as you've seen before, this command is supposed to install Emacs and Guile in a single transaction, right? So what's going on? So my goal here is to allow you to get a feel of what the interfaces are,
04:23
what they allow you to do, and if you're a seasoned skin programmer, maybe to allow you to go further. So first thing, so don't hesitate to interrupt me when I'm making demos and examples. We'll see how it goes.
04:41
So first thing you might want to be able to do, when you look at this thing, is how do you actually access packages themselves, and how do you look at a package? So I should have mentioned before, but the way it all started, I mean, the way Geeks started, is that I was a Nix user before and a Nix developer.
05:03
And I was really happy with Nix because it's really awesome. It does a great job. I mean, that's how functional package management got started, right? But I was a bit frustrated because as soon as I wanted to write a tool to fiddle with packages and stuff, it was really hard. And the reason for this is that Nix uses
05:20
an external domain-specific language. So at one point, for instance, I wanted to write an updater, a tool that would automatically update recipes for GNU packages inside Nix. But to do that, I had to use the Nix exporter, which would export the Nix syntax tree to XML, right?
05:42
So all the distribution to XML, and then to parse that XML to just extract a bunch of packages so that I could actually do that task, right? So that's pretty tedious and inefficient and everything. So I thought, wow, it would be great if we could just guide directly to do that.
06:01
So let me give an example. So I'm starting a Guile repo. Can you read it? I think so. So the first thing I can do from there is just import geeks. So I import a couple of modules.
06:23
And I'm just saying, I want to use these two modules, these two libraries. And these two libraries are actually geeks and the associated distributions, right? So we have just a normal scheme library, which is cool. So for instance, I can use this module.
06:46
And if I type coreutils, so I asked for the value of coreutils. It's just a plain scheme variable defined in GNU packages base. And I see that I have a package object. And that's it, you know, nothing special, OK?
07:01
So obviously, I can query the package. Is coreutils a package? Yes, it is. What's its name? You know, that kind of thing. Fine, but I can also obviously look at the package by name. Like I can do what's the package corresponding
07:24
to this specification. And I'm getting a package object. And this is what I would type on the command line, right? And that's how the command line translates specifications to actual package objects. And I can say, you know, like, guile 1.8.
07:44
Hey, it's not working. Well, you get the idea, right? So obviously, I can do things like iterate on the whole list of packages.
08:01
So we have a function which is called fold packages. And you give it, oh, we can read it here. It's below the screen. But anyway, we can give it a procedure and an initial value. And it will construct a list of all the packages, right? So here, it's telling me I have 3,000 packages
08:21
in the distribution, for instance. Yeah, and yeah, I think that's about it. Which one is it? OK. So with this in place, we can already write a bunch of pretty useful applications.
08:42
I mean, if you go to the Geeks website, there's an HTML page that contains a list of all the packages. And it's just trivially implemented just by using these helper functions, right? We can just browse the list of packages, generate XML from it, and we're done. And we have also several user interfaces.
09:02
So we have the command line interface, which you've seen before. We have an Emacs interface. And we have a web-based user interface, which is still under development. But already, just because it's just a scheme library, you can already write all these things, which is pretty cool, I think.
09:23
Any questions so far? OK. So now I need to talk about the store. So you've seen before in Ricardo's presentation that everything that we build with Geeks ends up in GNU store, right, in this special directory.
09:40
All access to that store is mediated by a daemon. So what happens is we have scheme code here, which is like the Geeks command, for instance, or my Guile repo. And it has a bunch of libraries loaded. And when I build something, like when I type geeks-package-i Emacs, and it actually
10:01
needs to build or to download Emacs, then that Geeks command here is going to talk to the daemon and ask it, well, can you build it for me? Or if it's already built, just tell me what this lengthy file name is. And if it needs to build it, then it's going to launch whatever needs to be done to actually build
10:23
the package. So it's going to run, configure, make, install, and so on, for Emacs plus all its dependencies. But this is all being done by the daemon itself on behalf of clients. This is so that we get isolations, right? So every package is built in an isolated environment
10:42
in a CH root under separate namespaces, separate UIDs, and so on. And so, yeah, that's how it works. The daemon itself runs as root. And we have our guy thing running here. So from Scheme, if we want to talk to the daemon,
11:02
well, we first need to open a connection, right? So this is my connection to the build daemon. And I can do basically RPCs, so remote procedure calls,
11:20
to the daemon to ask it to do various stuff. Yes? Why do you need a daemon? Why is it not just a library? So we need a daemon because to perform isolated builds, the daemon need to be able to make a CH root call and to set up separate namespaces, Linux namespaces,
11:41
and that kind of thing. And so, yeah, so far it has to. Could it be a daemon? No, because you need root permission to do that. And so we could use user namespaces, as was discussed before, but it's still a bit in flux in Linux. It's not their security issues and all that,
12:02
so we're not there yet. Yes, so for example, one of the RPCs that we have here is add text to store, which says, I want to add a file in the store. And users do not have direct access to the store.
12:21
So they have to rely on the daemon to do it on their behalf, right? So what I can do is that I can say, OK, daemon, OK, I can do that. And this is making an RPC, a remote procedure call to the daemon.
12:42
And the daemon, as a result, gives me the name of the file which has just been created. If I open the file, yeah, it's just, yeah, it created the file. So I'm not going to go into details.
13:00
There are many RPCs I can say. I can ask the daemon, is this a valid file in the store? And it's going to tell me if it's valid or not. That kind of stuff. Yeah, so that's basically, that's one of the main ideas. So fine, but if we're looking at what the daemon
13:21
provides, it's very, very low level. The daemon itself doesn't understand scheme at all, right? It doesn't know about configure, make, make install. So at some point we have to translate from our nice high level package definitions to something low level, I mean, instruction that the daemon can understand
13:40
and actually perform. So how does that work? Well, it's pretty useful to use. So this is a DAG that was generated with the geeks graph command. And it's showing the DAG, the graph of package objects for core utils. So this one is pretty nice, simple graph.
14:04
And it shows really the relations between package objects. But if you look carefully, you're probably wondering, well, something's missing here, right? Can anyone spot it? How do we compile all this stuff?
14:21
Right, so we need a compiler, we need Lipsy, we need grep, a bunch of stuff, right? And so we can actually look a bit further down. So there's a lower level abstraction in geeks. It's called bags. And in bags, a bag is a bit like a package,
14:41
except that we make all the inputs explicit, right? So here at the bottom we have GCC, Lipsy, set, grep, find utils, and so on and so forth. And we're just making it explicit here. Okay? But there is still something missing. Can you spot it?
15:02
Well, it's the same story again. How do we compile this, right? So, guess what? Yeah. We can actually ask geeks to show also the implicit inputs of those base packages, right?
15:24
And so at the bottom here, we have a bunch of bootstrap binaries which are pre-built, because we have to start from something, right? So if you attended Manoli's talk this morning, that's the whole story. We have to start from something, and this is captured by geeks, and it's here.
15:44
But it's still scheme, right? When talking about bags, we're still on the scheme side of things. So we are still not at the level of the daemon. And what the daemon understands is derivations, which are low-level build instructions. So it's just, basically, you give it an executable
16:01
and a list of arguments, and that's it. That's all it can do for you, right? And so if you try to plot the derivations, it's too big. Yeah, there are many more nodes and so on. So, how do we get there?
16:22
Well, so I have my coreutils package here, and I have my connection to the daemon, and I can say, okay, what's the derivation for coreutils? It's talking to the daemon, well,
16:41
it's computing the derivation, sending it to the daemon, the daemon validates it, and we get that as a result. So this is a derivation object. And we see two file names here, one is a .drv, which represents the build instructions, and the other one is the result, right? So what this is telling us here is that we have the .drv derivation, which are instructions,
17:04
and this is what we will get if we actually build this derivation. So if we look at the .drv, it's basically a text file that lists all the inputs of the build process, everything that needs to be visible in the chroot, right?
17:20
And somewhere here, there's a call to the guile executable that will actually run, configure, make, make install. And then we can ask the daemon to actually build it if necessary. This is seven. Okay, and it's, yeah, it's going to build it, trust me.
17:46
Okay, questions? So yeah, so just to summarize, we have several levels of abstractions. We have packages, bags, and derivations. And we also have other high-level objects that can be compiled into derivations, such as origins.
18:04
So if I go to, oh, wait. So if I look at my core.utils definitions, definition, it looks like this, define, blah, blah, blah, package, and so on.
18:23
And there is this origin thing here which represents a source code. And it's also something that can be compiled down to a derivation, right? So the derivation for the origin is going to download, well, the core.utils source code, okay?
18:43
Okay, and to finish, something a bit funny, I think, interesting. So if you remember this picture, you've probably noticed that there is guile pretty much everywhere except here. So this is written in C++. This is a daemon coming directly from Nix, right?
19:04
We're using the same daemon. But we have skim here and skim there, right? So there's a pattern, right? I mean, we wanna be able from here to describe code
19:21
that is going to be run there, right? So how can we do that? Well, Geeks has G expressions to do that. So anyone familiar with skim here? What's a fraction of people familiar with skim? Okay, most of the audience.
19:41
So if you're familiar with skim and Lisp in general, you probably know that it's possible in Lisp to have quoted data structures. So I can say, quote, MQDR, foo, let's say. And this is an expression, and I could write that expression to a file
20:00
and give that file to the daemon so that it runs MQDR, right? The problem for us in Geeks is that usually it's not just foo, right? It's a long file name, and we don't want to write it by hand. So we need a solution.
20:21
So the solution is G expressions. Let me give an example. So instead of backquote here, I'm going to use this hash tilde thing, okay? But it's similar in spirit to backquote. And you can say, let's say for instance, I want to, yeah,
20:40
I want to skim link core details. So I can write something like this. And this is, so you can think of hash tilde as backquote and hash dollar as unquote, comma, right?
21:00
And this is just producing a G expression which will, which represents, you know, there's a call to sim link with core details and an argument and output as a second argument. So output is a special argument. So you know that core details is a normal variable, as we've seen before, but output is not a variable defined here.
21:22
It's a special value, which means this is the output of the derivation I'm talking about, okay? So what's different compared to quote? Well, if we go to the, if we look at the internals of G expressions, well, we can say,
21:44
we can query the inputs, oh, wait, yeah. We can query the inputs of that expression. And as you can see, the G expression embeds information about what its dependencies are. And that's how it works.
22:02
And then we can convert it to, I'm not going to talk about monads today, but it's a topic for another day. I can say, okay, I have this G expression and I want to convert it to an S expression.
22:22
And that's what I get as a result, right? So the file name has been automatically introduced inside the G expression. So from there, I can make a derivation. I say, Gx2Derivation,
22:42
foo, and sim link, coreutils, output. Oh, so I'm not going to talk about monads, but yeah.
23:02
Okay, so if I use Gx2Derivation, I'm getting, as a result, a derivation again. And I can, again, have it built like this, and this is it. And if I look at the result,
23:21
well, it's, yeah, you can see it here. It's really a sim link, as expected, right? So this is a low-level mechanism that we have to have what we call multi-tier programming because we have scheme in different places and we need to be able to describe scheme code,
23:41
which is running in a different process at a different point in time, and referring to items in the store. Any questions? Okay, so just very briefly, I'm going to say that this is going further.
24:02
Obviously, I won't have time to do it, but this is going down to the level of the operating system declarations. So, as an example, this is the operating system declaration for my machine. And I can say,
24:21
where's my repo? I can load it from here. Okay, and I have an operating system here.
24:43
And again, I can build my operating system. Who is, I think it's called operating system to derivation. Is it not operating system derivation? Yes, and again, same story. It's evaluating the definition
25:01
for my complete operating system and returning a derivation that contains everything, right? From, you know, system service configuration, initial RAM disk, everything. Okay, and yeah, this is it. Any questions?
25:21
So, did you say the Nix, given Nix is a successor of Nix or a better one, but is it a successor of Nix OS? Yeah, I wouldn't call it a successor. But yeah, so Nix is comparable to Nix,
25:41
so it's package management. And then we have Nix SD, which is comparable to Nix OS, I mean, in terms of functionality. But obviously, Nix OS is much more mature than Gigs SD, and you know, so there are differences. But these are different approaches, both technically and also in terms of goals.
26:00
Like we're strongly committed to supporting free software, whereas Nix OS is much more lax in that respect. So these are different projects, I would say, but obviously with a lot in common. Yes? Okay, so this isn't quite related to this talk, but this is the second year's talk, and I think that's a good question.
26:21
And that's runtime. So it's all more or less clear when it's command line, so you just execute it anytime. But what if I have two different versions of some demo, like database, and they need to run on different ports so they don't collide? And what if I want, just like if I enter one environment
26:43
or whatever it's called, so can I make configure, you know, make it so that if I run, I don't know, like PSQL in one environment, it will automatically like hit one port and one version of Postgres,
27:02
and if I do it in different environment, it will hit the port or something like that. So what's the story with, like, where are the, you know, data paths, right? Yeah. Different versions of demos that run at the same time,
27:22
and how do they work? I see. Do they separate? Yeah. On the path? So I didn't talk much about GeeksSD itself, but basically when using GeeksSD, you provide a specification of your operating system. So that includes everything, you know, all these details here.
27:42
And one of the important parts is services, system services. So you have a services field, and you just give it a list of service objects. And if you look at this particular list, so this is one that we provide by default for people who want to run like XFC, stuff like that.
28:01
And we're just building up a list, right? With screen lockers, UDISC, Avaei, all that. So if you want to have two different PostgresQL services, for instance, you would just call, I think it's called Postgres, probably Postgres service, basically,
28:20
specifying different options. So you would say this one is going to listen on this port and this one is going to listen on that port. And under the hood, it's going to generate configuration files, command lines, everything, such that when you run it, it's just doing what you want it to do, right? Okay? Okay.
28:41
Thank you.