nix-processmgmt: An experimental Nix-based process manager-agnostic framework
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 19 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/50708 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
Nixcon 202010 / 19
2
3
7
8
12
14
15
17
18
19
00:00
Data managementInformationPresentation of a groupProcess (computing)BitFerry CorstenSoftware frameworkIntegrated development environmentSoftware developerAndroid (robot)Rootkit
00:54
Process (computing)Software frameworkSubstitute goodSource codeRevision controlDeterminismFreewareSystem programmingOperations researchCodeHash functionIntegrated development environmentInstallation artGastropod shellVertex (graph theory)Web serviceExecution unitDemonComputer fileDistribution (mathematics)Augmented realityComputer-generated imageryNamespaceSoftwareFormal languageFunction (mathematics)Scripting languageAttribute grammarLocal ringComputer configurationOrder (biology)Projective planeExecution unitElectric generatorSystem programmingCASE <Informatik>Process (computing)Data managementCombinational logicBuildingField (computer science)Web serviceDistribution (mathematics)Installation artMedical imagingExtension (kinesiology)Constructor (object-oriented programming)Self-organizationConfiguration spaceExpressionRevision controlFunctional (mathematics)Right angleProxy serverInstance (computer science)VirtualizationNamespaceKey (cryptography)Portable communications deviceCategory of beingVariety (linguistics)EmailFormal languageModel theoryWindowNumberSoftware frameworkOperating systemDiagramFluid staticsIdentity managementWeb pageSource codePresentation of a groupOperating systemWeb applicationAdditionBinary codeCache (computing)MultiplicationVirtual machineServer (computing)Gastropod shellWeb 2.0Utility softwareSoftware developerAttribute grammarBit rateSpacetimeHand fanDeterminantNatural languageCartesian coordinate systemMusical ensembleMachine visionKeyboard shortcutScripting languageData storage deviceAngleMobile WebNetwork topologySampling (statistics)SurfaceInformationPower (physics)Video gameCycle (graph theory)Limit (category theory)Buffer overflowSuite (music)XMLComputer animation
09:43
Local ringSystem programmingIntegrated development environmentLevel (video gaming)Web applicationProcess (computing)DemonExecution unitFunction (mathematics)Scripting languageSoftware frameworkLatent heatAbstractionWeb serviceData typeAugmented realityMathematical optimizationParameter (computer programming)Instance (computer science)Principal ideal domainInheritance (object-oriented programming)Uniqueness quantificationEmailInterior (topology)Constructor (object-oriented programming)Virtual realityInstance (computer science)Latent heatScripting languageWeb serviceDemonProcess (computing)Line (geometry)System programmingMobile WebSoftware frameworkFunctional (mathematics)Combinational logicData managementWhiteboardVirtual machineNatural languageSurfaceMixed realityCategory of beingDemosceneExpressionFunctional (mathematics)Execution unitState of matterAbstractionWeb browserField (computer science)Cartesian coordinate systemLimit (category theory)Server (computing)Information managementWebsiteLevel (video gaming)Parameter (computer programming)BitEscape characterConstructor (object-oriented programming)Sheaf (mathematics)Uniqueness quantificationIntegrated development environmentBootingHardy spaceShared memoryResultantVotingStaff (military)NumberData structureTranslation (relic)Procedural programmingMultiplicationComputer fileComputer programmingDifferent (Kate Ryan album)Gastropod shellGroup actionRadio-frequency identificationOrder (biology)Metropolitan area networkPerspective (visual)Computer virusDean numberSet (mathematics)Position operatorModel theory2 (number)AdditionKeyboard shortcutUniform resource locatorWritingoutputAsynchronous Transfer ModeElectric generatorRadical (chemistry)Pattern languageWeb 2.0Configuration spacePrincipal ideal domainProxy serverGeneric programmingEmailWeb applicationComputer animation
18:32
Constructor (object-oriented programming)System programmingInstance (computer science)Entire functionScripting languageProcess (computing)Web applicationLatent heatData managementLine (geometry)Dependent and independent variablesMedical imaging2 (number)Graph coloringSource code
19:41
Process (computing)Constructor (object-oriented programming)Inheritance (object-oriented programming)Web applicationInstance (computer science)Virtual realityEmailSystem programmingDemo (music)Reverse engineeringOpen setProxy serverScripting languageComputer networkMaß <Mathematik>Computer programFreewareSystem programmingProcess (computing)Data managementMultiplicationFreewareWeb serviceCombinational logicMultiplication signScripting languageInstance (computer science)Computer animation
20:05
Group actionUniqueness quantificationComputing platformProcess (computing)Function (mathematics)UDP <Protokoll>Java appletWeb serviceTomcat <Programm>Server (computing)DatabaseApache <Programm>Intrusion detection systemGroup actionData managementJava appletVirtual machineComputer networkWeb applicationInstance (computer science)Process (computing)Software frameworkSystem programmingNumberWeb serviceProjective planeMessage passingAreaPresentation of a groupActive contour modelComputer animation
21:16
BlogProcess (computing)Self-organizationDeclarative programmingWeb serviceSoftware frameworkFunctional (mathematics)Translation (relic)Module (mathematics)Instance (computer science)Latent heatMultiplicationProjective planeExecution unitLevel (video gaming)Process (computing)Web serviceConstructor (object-oriented programming)AbstractionSoftware frameworkSource codeShared memorySoftware bugSystem programmingData managementModel theoryComputer animation
22:13
Process (computing)DemonIntegrated development environmentPrincipal ideal domainFunction (mathematics)Interior (topology)Parameter (computer programming)EmailUniqueness quantificationInstance (computer science)PRINCE2Web applicationModulare ProgrammierungModel theoryModule (mathematics)System programmingWebsiteFront and back endsNetwork topologyWeb serviceNetwork socketQuicksortProcess (computing)Communications protocolMultiplication signProper mapChromosomal crossoverData managementMultiplicationSoftware frameworkLatent heatFluid staticsDifferent (Kate Ryan album)Formal languageFunction (mathematics)Derivation (linguistics)BuildingRun time (program lifecycle phase)Electric generatorGeneric programmingTheorySet (mathematics)MeasurementGroup actionTranslation (relic)Presentation of a groupComputer animation
Transcript: English(auto-generated)
00:00
Our next talk is Nix Process Management, an experimental Nix-based process manager agnostic framework, and the presenter is named Sander van der Berg. This talk is about complementing Nix with any process manager. A little bit more information about Sander, if you're not familiar, is that he is a Nix contributor since 2007,
00:22
who's worked on so many things such as the FHS root environments, and the Nix Android build environment. He is the main developer for various Nix-related utilities, such as DisNix, Dysnomia, Node2Nix, and Composer2Nix. I would also suspect that a lot of Nix OS users
00:43
have probably encountered Sander's blog, which is sandervanderberg.blogspot.com. With that out of the way, you can free to start, Sander. Thank you for the kind introduction. Hello, everybody. This presentation is going to be about
01:01
a personal research project that I've been working on in the last year or so. I think it addresses a very important shortcoming of basically the tools in the Nix ecosystem. As you may probably already know, the Nix package manager is a powerful solution.
01:21
It offers all kinds of nice features, like you can conveniently construct packages from source code and all the required build time dependencies. It offers build determinism, transparent binary deployments by downloading existing builds from a binary cache. It allows you to store multiple versions and variants
01:43
of the same packets safely next to each other. You can do unprivileged user deployments, so if you want to install packages, you don't need to be root. It can also be used on multiple operating systems. In addition to Linux, it is also well supported on Mac OS,
02:01
and very recently, FreeBSD was also accepted into the FreeBSD ports tree. With some little effort, you can also use it on other Unix-like operating systems as well. When I have to explain Nix to newcomers, what I typically use is a Nix shell example to
02:22
show all nice interesting properties of Nix. For example, if you're on a conventional Linux distribution and you want to install with packages like Python or Node.js, you may already have a version of Python installed on your machine and you may not have installed Node.js on your machine yet.
02:42
What you can conveniently do is you can spawn a shell session in which you have a certain version of Python and Node.js, and the Nix package manager will automatically install it, and in this shell session, you can basically use these development utilities. The nice thing is because they're stored in a Nix store, they will not conflict with other versions of packages,
03:02
and they won't interfere with packages installed on your host system either. When I show this to people, especially newcomers, they typically get quite happy. Basically, what they tell me is, yes, this is exactly what I'm looking for, so I'm going to experiment with packages.
03:21
I'll try to install PostgreSQL or the Apache web server or Nginx because, yeah, Nix is a very convenient tool to experiment with packages, and then I have to disappoint people because I have to explain them. Nix is a package manager, it's not a service manager.
03:40
You can install, for example, the Apache web servers on your machine, but it just provides you the executable. You're still responsible for configuring the service yourself, and also make sure that the lifecycle of the process gets managed. In order to do that, you basically need to use some other tool like System D,
04:01
and that is actually quite confusing for newcomers. What I also typically do is I explain newcomers that there's, of course, the Nix package manager, but there are also sister projects that can complement Nix with all kinds of other deployment features, such as process management. I think the most famous project is Nix OS.
04:22
Nix OS can generate System D unit files with the package manager, and then System D is responsible for managing the lifecycle of these processes. But there's one catch, if you want to use Nix OS, that basically means that you're forced to adapt a fully Nix-managed Linux distribution.
04:41
If you want to use another operating system or if you're not familiar to Nix yet, this is typically a pain. For seasoned users like me, this is not a big deal. This is actually what I want, but it's not always an option to use Nix OS. The same thing, for example, applies to macOS.
05:00
On macOS, there is a project called Nix Darwin that offers all kinds of system services that you can manage with LaunchD. But obviously, it's limited to macOS only. People typically tell me when they're on conventional Linux distributions, yeah, perhaps I should use Docker because, yeah,
05:21
with Docker, I can manage services. Then I can, of course, also tell people that Nix is also an interesting solution to use in combination with Docker. You can basically use Nix in the construction of images. You can even use Nix to build Docker images completely, and these images are typically way more
05:41
space-efficient than conventional Docker images. But this is not always a compelling use case because what they're basically doing is we squeeze Nix into a Docker workflow, but it's not a Nix-driven deployment process. This is basically my motivation to start developing the next process management framework.
06:02
It's basically a general solution for complementing Nix with process management facilities, and it's built around a number of key concepts. First of all, it's entirely Nix-driven, so the idea is that you write system configurations completely in the Nix expression language.
06:21
It's also based on simple conventions. That means you follow the similar conventions to how packages are organized in the Nix packages repository. So the idea is that for running processes, you write a function definition, and to compose running process instances,
06:40
you define an attribute set with function invocations. And a nice small extension to this framework is that you can also organize process dependencies with the same formalism as well. And the framework will automatically arrange the ordering if needed, so that, for example, processes are activated in the right order.
07:03
Another key concept is that it's process manager agnostic, so it's not designed for a specific solution, but it should work with all kinds of process managers. So currently you can use System 5 init scripts, Supervisor D, System D, Launch D, BSD RC scripts,
07:22
and Windows services on Sigmund. But the nice thing is the model is flexible enough that you can even use it with solutions that are not qualified as process managers. So you can, for example, also use it in combination with Disney and Docker. And the reason why you can do that is these solutions are multifunctional solutions
07:41
and they can also organize processes. So in the framework, we can basically use these properties for basically all kinds of interesting reasons. It's also operating system agnostic because yeah, it supports process managers on a variety of operating systems and Nix is portable to some degree as well.
08:04
You can also use it for unprivileged user installations. And the reason why that is possible is because I built in a global switch that allows you to disable the creation of users and changing user permission. So as a non-root user, you typically don't have the permissions to do this.
08:21
And by disabling it, you can basically just run any process you want without restrictions. And the final key concept is that it doesn't require any advanced concepts like namespaces and cgroups that are commonly used for containers. So the solution relies on conflict avoidance rather than isolation.
08:41
And that is for example, good for portability as I'll explain later in this presentation. So to explain you how the framework works, I developed a very simple example system. It's actually quite an over-engineered example,
09:00
but I think it's quite easy to understand how it works. So basically this is a web application system that consists of multiple running processes. What you see on the right of this diagram are three web application processes. So they're basically processes with an embedded HTTP server.
09:21
And the only purpose of these web applications is to render a static HTML page that basically states the identity of the service. In front of the web application process, there's Nginx that acts as a reverse proxy. And Nginx basically redirects users
09:40
to the web application process instances based on the virtual host header fields. So for example, if the user in the web browser opens the URL webapp1.local, then Nginx will redirect the user to the first web application instance. Likewise, if you use webapp2.local,
10:00
then the user gets redirected to the second web app instance. So in order to automate the deployment of the system, so we have four process instances, you need to write Nix expressions for each process instance. And one way of doing that is by specifically writing Nix expressions
10:22
for a process and a specific service manager, such as system five in it. And this is basically what a Nix expression looks like. So as you may probably notice, this is actually quite similar to the convention of how we in the Nix package manager declare packages.
10:41
So this is a function definition. The first line is basically the function header. And these refer to all the built inputs that are required to generate a system five init scripts. So create system five init scripts is an abstraction function that allows you to basically generate a huge shell script
11:02
with deployment activities. Web app refers to the web app application executable and port refers to the TCP port number that the server should bind to. And what I do in the body is I invoke the function abstraction and I basically specify what all the deployment activities should look like.
11:21
So this, for example, is the start activity. It starts the executable and it uses the dash D parameter to specify that it should run in daemon mode. The stop activity is stopping the executable. The restart activity is basically calling stop and start. And status is basically used to basically show
11:42
whether the process is running or not. The run levels parameter is to specify the run level. So I don't know if you still recall system five init scripts that we used to develop 10 years ago. So when you boot in run level three, that is typically used to boot in terminal mode
12:01
and run level five is used to boot a graphical desktop environment. And basically this parameter states that the service should start on boot up for all these three run levels. Now writing a system five init script is a bit for both. So if I have to, for example,
12:20
package another process like nginx, then I end up repeating the same patterns over and over again. So what I also did is I developed a higher level abstraction function for system five init scripts that looked like this. So instead of specifying the activities, what I do is I specify the process that I want to manage. So this is the executable I want to manage
12:42
with this command line parameter. And basically what the function abstraction does is it infers the activities automatically. So basically the result of evaluating this function invocation is this. So the framework also has abstraction functions for other service managers.
13:02
So this is system five init, but maybe I want to use system D instead. So this is basically the function abstraction that you can use to generate system D unit configuration files. So again, what I do here is I basically, I basically specify here what executable I want to manage and to which TCP port it should bind.
13:22
And basically what this function abstraction does is it generates a system D unit configuration file with nearly an identical structure. So this is pretty much a one-on-one translation from a nyx expression to a system D unit configuration file. And the framework is basically full
13:41
of other abstraction functions for all kinds of process managers. So there's also, for example, the create supervisor D program abstraction function. There's an abstraction function for launch D, for BSD or scripts, and many more. Now, the interesting thing is if I would compare, for example, the system D expression
14:02
with the system five expression that I've shown in the previous slide, then you see that they're slightly different, but they're not all that different. They're still mostly the same. So what I also did in the framework is I created an abstraction function that abstracts over all these process manager
14:21
specific abstraction functions. And that basically looks like this. So what I do in this particular nyx expression is I use a generic create manage process function that basically describes from a high level perspective what process do I want to manage. And these concepts can be easily translated to function implications
14:41
to the target specific abstraction functions. And basically any process manager in the framework is supported. So system five, launch D, system D, they can all work with a high level specification like this. The only thing that you need to do is
15:01
system D prefers to work with processes that run in the foreground, whereas system five in it wants processes to demonize on their own. What you need to specify is basically for both foreground processes and demon processes, what kind of additional settings do they require?
15:21
So this demon arcs parameter basically specifies if the process needs to demonize, then pass these common line arguments to the executable. And if you want to run as a foreground process, basically nothing is required. Actually specifying foreground process and demons is not a strict requirement. They can also be simulated,
15:41
but for an optimal user experience, it's important that you still make a distinction between those two. Then there's one more concept that I need to explain. So in the example that I've shown you earlier, we want to run web app processes,
16:02
but we actually need multiple instances of them. We need three of them. To be able to construct multiple web application instances, you need to follow a slightly different convention. So basically what I do here is I declare a nested function and the outer function are basically, so the first line is referring to parameters
16:20
that apply to all web app process instances. And the inner function header refers to parameters that are instance specific. And if you specify a unique combination of these parameters, then multiple instances can coexist on the same machine. So for example, the port number, only one service can bind to a specific port number.
16:44
But if you allow every instance to bind to a unique port number, they can coexist. And the same thing applies to the instance name. So normally when you launch a process as a demon file, the demon, the bit file has the same name as the demon.
17:02
But if you want to run multiple instances, you have to generate PID files with unique names. And basically the instance name parameter is a high level concept that allows you to basically generate unique PID file names so that process can coexist.
17:22
So in addition to declaring constructor functions that you can use to create process instances, you also need to actually specify what process instances you need. And that is done in a composition model that looks like this. Again, this is similar to the top level expression
17:41
in the next package's collection. Again, this declares a function. So these are parameters that basically apply to all. Excuse me, Sander, sorry to interrupt you. Yes. You're in like, your Q&A portion is in about three minutes, so you'll be eating into that.
18:02
Oh, yeah, I'm almost done. Yeah, so these properties apply to all running instances. And what I do here is here I construct two web instances. They can coexist because they have a unique port number and instance suffix. And what I'll do in this expression as well,
18:22
I construct an nginx reverse proxy. And yeah, that is basically responsible for setting up the redirection. So I'm actually gonna take a risk now. I'll show you that this is basically the example.
18:42
And I can basically deploy it as any. So this basically deploys the entire system as a collection of system five in its scripts. And as you can see, it deploys five instances of the web application and two nginx instances. And this basically shows that the redirection is working.
19:03
So now the first instance is responding and now the second instance is responding. And I can also deploy and deploy the system like this. And now it's not running anymore. I can also deploy it as systemd scripts. So if I change the command line instruction
19:22
from system five in it to systemd, I can deploy the entire system as systemd in its scripts. And as you can see, the system responds again. And if I request the process overview, then you can also see that the system is running. So I basically use one single specification
19:42
to deploy the system for multiple process managers. So I'm almost done. So I'll just wrap up because I'm running out of time. There are lots of more interesting combinations possible. So you can also deploy in FreeBSD as bsdrc scripts. You can even deploy docker containers
20:01
for the process instances. Yeah, there are lots of other features possible that I haven't explained. So you can also basically create users and groups so that they processes run as unprivileged users. You can automatically assign port numbers, user IDs and group IDs.
20:23
You can also combine the Nix process management with framework with this Nix. And the nice thing is that you can deploy process to networks of machines and you can also combine them with things that aren't processes. So for example, Apache Tomcat can be managed as a process,
20:42
but you can also deploy Java web applications to that running Tomcat instance. So this is very nice to deploy very heterogeneous systems. Okay, thank you, Sander. Is that the end of your presentation? Yeah, it's pretty much done.
21:01
So yeah, there's some future work and if you want to play around with this link. And yeah, that's pretty much it, so. Okay, so we have about, I think, five questions. I will start with the first one. This one is from Nikku on IRC. And they said, do you think that NixOS, Nix Darwin
21:20
and other projects managing services around Nix packages could in the near future move to such an abstraction layer and share more source definitions, et cetera? I think that is very well possible, but yeah, the biggest problem we have to overcome that is related to Ilko's discussion from yesterday is
21:42
the module system, for example, is somewhat problematic if you want to support features like being able to construct multiple instances of processes, because modules are basically not units of instantiation. So that is a problem.
22:01
There are actually concepts that you can integrate into the framework. So for example, the translation process from a high level specification to any process manager, that is actually also something you could integrate in the module system. Basically the concepts are, I think, very generic.
22:22
So what we could also in theory do in Nix packages is we have a system D layer and perhaps a launch D layer, and we also implement a module that resides somewhere in the middle of the translation process. So a generic process management layer. And if we use that, then we can already
22:41
with the same specifications, support multiple process managers. Of course, another small thing we need to do is we need to be able to use this service layer separately, but I think that is something we can easily overcome. So. Awesome. All this is actually really interesting to me.
23:00
I was like, super surprised with your presentation. Yeah. Yeah. So another question we have is how do you envision to generalize some system D specific features like hardening or socket activation that some services use? Yeah, that's a good question. So socket activation, for example,
23:21
that is a concept that also launched D supports, but the sad thing is, is that they don't follow the same protocol. So it's fair. I don't think it's impossible to generalize that, but I think it's very difficult. So what you can of course still do is you can define overwrites for the process managers
23:43
that you want. So you can basically, if some service needs socket activation, then you can, for example, declare for launch D, do this and for system D do something else. Yeah, sadly you can't use a generalized concept, but it's still possible to, yeah,
24:02
to fill the, yeah, to address these deficiencies with overwrites, if you want to. Great, great. And I think we have one more question, pretty short one, I think. So by running processes as different user and group, do you mean running the service as a proper system service? So not a user service anymore.
24:22
Yeah, yeah, that's exactly what I mean, yeah. Okay, and another one, which I'm not sure you'll have time to completely finish, but we'll give you a minute, is what about a static process manager? EG, doing a topological sort at next build time to output a certain derivation representing the processes, which will run at runtime.
24:43
I'm not sure if I completely understood the concept of static process manager, but that is actually already possible if you combine this framework with Disneyx as a process management backend.
25:01
It uses the next language to basically generate a dependency tree. And the only thing Disneyx does is it invokes a module that simply starts the process. So that is already somewhat possible. So, but yeah, the framework is not really designed
25:21
to statically generate dependency trees itself. It only tries to utilize the features of the process manager backends as well as it could possibly do. Okay, great. Thank you so much. We are now out of time for the Q&A portion.
25:42
There's some like really interesting sort of crossover between the talks happening here, I think, which is pretty cool. So yeah, thanks. And everyone please remember to, you know, put those clapping emojis in the IRC chat and show the love. Yeah, and the breakout room for this talk, since it seems to be pretty lively Q&A going on here
26:00
is nix-process-management and management being MGMT, like an acronym. Yep, okay everyone. See you all in five minutes.