Using Nix to build development environments as you always wanted
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 266 | |
| Author | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/66587 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSS4G Prizren Kosovo 202318 / 266
10
17
23
44
45
46
47
48
49
50
52
53
80
84
85
91
110
116
129
148
164
167
169
173
174
181
182
183
186
187
199
202
204
206
209
215
241
248
265
00:00
Projective planeSlide ruleComputer animation
00:28
Open sourceFreewareSoftwareMultiplication signSoftwareStapeldateiComputer animation
00:59
Core dumpOrder (biology)Raster graphicsGraphical user interfaceGrass (card game)Library (computing)Web applicationProcess (computing)Different (Kate Ryan album)Projective planeOcean currentMathematicsMemory managementRevision controlSoftwareSoftware bugState of matterCASE <Informatik>Cartesian coordinate systemMultiplication signCore dumpLevel (video gaming)File formatGeometryVideo gameTransformation (genetics)DatabaseCycle (graph theory)Computer animation
03:02
Graph (mathematics)SoftwareCodeService (economics)Library (computing)Different (Kate Ryan album)Mixed realityMedical imagingSoftwareSource codeCore dumpDigitizingSoftware repositoryComputer animationLecture/Conference
03:52
SoftwareUbuntu <Programm>Library (computing)Electronic data processingVirtualizationSource codeDefault (computer science)Web serviceCore dumpComputer animationLecture/Conference
04:34
Computing platformRevision controlRevision controlLibrary (computing)Different (Kate Ryan album)SoftwareProjective planeFingerprintComputing platformNeuroinformatikCurvatureComputer animationLecture/Conference
05:28
NeuroinformatikSoftware1 (number)Connectivity (graph theory)Data managementComputer animation
05:59
Uniqueness quantificationComputing platformGastropod shellRevision controlSingle-precision floating-point formatInstallation artOcean currentCartesian coordinate systemFerry CorstenGastropod shellBitQuantum gravityComputer configurationSoftwareProfil (magazine)MultiplicationMobile appData managementMehrplatzsystemSoftware repositoryInternetworkingRevision controlRepository (publishing)Medical imagingCompilation albumDifferent (Kate Ryan album)Thermodynamisches SystemBuildingWordWindowDistribution (mathematics)Connectivity (graph theory)Function (mathematics)Electronic data processingComputer animation
08:33
Formal languageFunction (mathematics)Source codeRevision controlHash functionKey (cryptography)Endliche ModelltheorieFunctional (mathematics)NumberGeometryWritingSpecial unitary groupBuildingFormal languageConfiguration spaceSource codeContent (media)Parameter (computer programming)Module (mathematics)Core dumpHash functionFunctional programmingInternetworkingRevision controlComputer animationLecture/Conference
09:50
SoftwareNetwork topologyHill differential equationCore dumpConnectivity (graph theory)Memory managementDistribution (mathematics)SoftwareDifferent (Kate Ryan album)NumberFree variables and bound variablesRepository (publishing)Computing platformGraph (mathematics)BuildingFreewareComputer animation
11:51
Thermodynamisches SystemModul <Datentyp>SoftwareConfiguration spaceModule (mathematics)Declarative programmingService (economics)Server (computing)DatabaseOperating systemConnectivity (graph theory)Thermodynamisches SystemArithmetic progressionService (economics)NeuroinformatikDatabaseSoftwareOpen setUniqueness quantificationModulare ProgrammierungBackupComputer animationLecture/Conference
12:56
Source codeThermodynamisches SystemTwitterSoftwareMemory managementTouch typingInternetworkingRevision controlTerm (mathematics)CurveNumberRandomizationStatement (computer science)Video game consoleAuthorizationComputer animationLecture/Conference
14:22
Thermodynamisches SystemHypothesisModule (mathematics)EmailComputerData modelSoftwareFunctional (mathematics)BitVirtual machineComputer programmingModulare ProgrammierungHypothesisThermodynamisches SystemComputer animationLecture/Conference
15:11
Derivation (linguistics)Function (mathematics)outputProcess (computing)Hash functionInternetworkingRevision controlFile formatDirectory serviceContent (media)Algebraic closureBinary fileCache (computing)Local ringVirtual machineSoftwareIntegrated development environmentSuite (music)Point (geometry)Validity (statistics)outputDirectory serviceInternetworkingData structureHash functionLibrary (computing)DialectMultiplicationMultiplication signData storage deviceSoftwareFunction (mathematics)Natural numberSuite (music)Thermodynamisches SystemRevision controlService (economics)Software testingCache (computing)Graphics tabletCryptographyContent (media)CurvatureConfiguration spaceMemory managementVirtual machineBitCASE <Informatik>Integrated development environmentKey (cryptography)Different (Kate Ryan album)Flash memoryComputer animationLecture/Conference
19:36
System programmingDemonDefault (computer science)Installation artThermodynamisches SystemComputer animationLecture/Conference
20:01
Software repositoryComputing platformThermodynamisches SystemRevision controlGraphical user interfaceComputerSoftware repositoryMathematicsVirtual machineProjective planeMultiplication signPatch (Unix)Rollback (data management)Configuration spaceThermodynamisches SystemControl flowSlide ruleMoving averageWeb pageComputer animation
20:34
BuildingIntegrated development environmentProof theoryThermodynamisches SystemProjective planePoint (geometry)Service (economics)BuildingProof theoryIntegrated development environmentMedical imagingVirtualizationSoftwareWindowDistribution (mathematics)Presentation of a groupFunctional (mathematics)Computer animationLecture/Conference
21:25
Computer fileInteractive televisionGastropod shellDefault (computer science)Projective planeCommon Language InfrastructureCellular automatonConfiguration spaceComputer fileThermodynamisches SystemService (economics)Form (programming)Default (computer science)Functional (mathematics)Gastropod shellDifferent (Kate Ryan album)Computer animation
22:02
Integrated development environmentGraphical user interfaceThermodynamisches SystemCommon Language InfrastructureSoftwareGastropod shellDirectory serviceThermodynamisches SystemDefault (computer science)Projective planeDirectory serviceRootComputer animationLecture/Conference
22:35
Gastropod shellInteractive televisionDefault (computer science)HookingThermodynamisches SystemLevel (video gaming)Gastropod shellFunctional (mathematics)Cellular automatonServer (computing)Parameter (computer programming)Default (computer science)Web 2.0Directory serviceClient (computing)Configuration spaceDatabaseRevision controlWeb serviceSoftware developerMessage passingComputer animation
24:32
Process (computing)Gastropod shellData managementService (economics)Integrated development environmentService (economics)Process (computing)LoginComputer animation
24:53
Patch (Unix)Revision controlLetterpress printingOptical character recognitionRevision controlSoftwareCellular automatonProjective planeWeb pageConfiguration spaceMathematicsThermodynamisches SystemIntegrated development environmentPatch (Unix)Gastropod shellComputer animationLecture/Conference
25:55
outputRevision controlSimilarity (geometry)GeometryComputer animation
26:19
Service (economics)Configuration spaceService (economics)Memory managementModulare ProgrammierungSimilarity (geometry)Programming languageCuboidComputer animationLecture/Conference
26:56
Thermodynamisches SystemData managementConfiguration spaceModul <Datentyp>Declarative programmingModulare ProgrammierungConfiguration managementDeclarative programmingIntegrated development environmentConfiguration spaceLecture/ConferenceComputer animation
27:20
Uniform resource locatorIntegrated development environmentNetwork topologyComputer animation
Transcript: English(auto-generated)
00:08
Hi, I'm Ivan Mencik. I'm from Slovakia. I came to tell you something about Nix technology and ecosystem and How to use it for your geospatial project. It's a huge topic. I have 100 slides. I need to be
00:26
really quick Yeah, like I'm a long-term Linux user Geospatial software user I started contributing to Nix in 2021 and this year I started Nix packages or Nix
00:43
geospatial Team. Yeah, and if you want to know more about Nix, I have a specially customized badge This is how you can recognize me. This is Nick's logo. Just like come to me. There will be probably no time for questions Yeah, so like what's the problem with it's not the problem but this is the current state of geospatial software we have
01:06
multi level dependencies in our software stack we have core libraries which Don't depend on anything geospatial usually which is like pros and geos and then we have another level of core libraries
01:22
Which depend on those libraries and then we have a Python libraries which depend on what's below On those core libraries and then we have another level of Python libraries Which depend on those Python libraries and they depend on this and then on top of that we have databases which depend on Anything below and we have web applications. We have desktop applications
01:45
We have like AI applications and they depend on on whole stack and we totally depend on those core libraries to provide us access to To formats to transformations to
02:01
Some processing Processing pipelines and we are extremely happy if like new new format is added to to GL or Some bug is fixed but like the lower the change happens The bigger is the impact on anything which is On top of those libraries. So many times there's a bug fixed and we want to
02:25
Apply this straight away to whole our stack, but there are situations when even if it's a bug and we have a bug fix it's going to break our processing pipeline because our processing pipeline works that way and the Fix is going to break so we don't want to apply
02:43
immediately and like sometimes we can't even work with the latest version because it doesn't work for our use case and usually we have Heaps of projects we don't have a one project and all those projects have like different Requirements and different life cycle and different
03:02
Age so this is how many dependencies are coming to play to Python Fiona Library, this is the core C library and this is all of those dependencies you need to successfully run
03:21
Run Fiona, basically, it's really really a lot So like how we install our software, sometimes we compile from source code. Sometimes we get it from Linux Linux This repos we get it from PPAs ours flat pegs snap images container images and
03:43
Anything else so like it's a huge mix of software and different kinds of software come from different vendors and different different sources, so this is a typical Example you have been to Linux you install g.js approach like core libraries from Ubuntu GIS
04:04
Then for some kind of data processing you install a Fiona to your Python virtual and by the way Fiona comes with GDAL again It's optional but this is the default. This is the default you can turn it off but it comes with
04:21
GDAL bundled in the wheel for data processing We install a QGIS using flat pack and then we have some web service installed to our Alpine container So typical problems, which we usually have like Because everything comes from different vendors. We have totally inconsistent
04:44
libraries like we can have like five different gdals easily on our Computer one from flat peg one in From pypi and and so on we have a we are going to have a problems to replicate it on different
05:00
platforms like bunted gis doesn't usually work with arch Linux doesn't work with Mac OS we might have a huge problems to reproduce it in a next after five years from now and like The biggest problem is how to support like really dozens projects of different age and
05:23
with different requirement different software version different customizations So next really can solve that this is One of the main purpose why it was designed to really like Correctly run software on our computer. So what is next there are like
05:46
Many components of basic next ecosystem going to talk about the main ones for the main one one of the like the first thing is nix as a package manager the first component it is really
06:03
not Using these words for no purpose. It is really the most unique and Advanced package manager it is cross-platform Which means it runs on any Linux distribution it runs on Mac. It can run on Windows WSL 2 and
06:21
Outputs of nix are bit-to-bit Reproducible this is reproducibility is big topic more on that later So it can run multiple versions of software without dependency conflict. This is not very common Think nix doesn't conflict with other package managers or software on your whole system
06:42
So you can run your Ubuntu and up and you can still run nix easily and it has very strong cross compilation Capabilities it can build Container images it can build Data processing pipelines it can build operating system. It can build whatever
07:02
Basically you want so this is the example of single application Installation like traditional way traditionally looking way how to install software so next profile install GDAL from somewhere so this will install GDAL for the user executing the command
07:21
You can have multiple profiles multiple users with different things, but this is not the best way how to do is just like Example you can create some temporary on-demand Environments using different command called nix shell this will install GDAL as well
07:41
You can play with that do your work if you do exit you don't have GDAL Anymore you can run software from any arbitrary internet Repository using another command nix run There's a github repo my github repo and it's running qg's a long-term release
08:04
Just with option to print the version You see before I was not able to access after I was not able to access qg's So it's just like example of just running software without without Anything else? This is the example using old
08:22
old command line interface how to run on Current one two words and how to run GDAL 1.7. No problem to do it so to Describe
08:41
to write build Instructions and Everything at nix we are using nix language. It is pure lazy evaluate the functional language this means something I'm not going to dig into it too much We use it to write build instructions, which means packages modules functions
09:04
And you can think about it as a JSON with functions. It is really like a good mental model for starting this is the example of of Basic build instructions for for geo so we have We have a function which takes
09:23
Few arguments like the name version and then we have another function with download the source code from from internet it has exact hash of the content of the source code and then we have Q Key dependencies you make and Nix will automatically build GDAL from from this instruction
09:44
It's not going to be like the best possible build configuration under the sun, but this is the basic Example of it. So another core component of nix ecosystem is nix packages Nix packages is really the largest and most up-to-date collection of packages on our planet
10:03
We build packages for Linux and makes or make OS on different platforms Nix packages contains free and non free software a lot of non free software, which is not built but But you can use nix install this Non free software just just to be compliant with licenses
10:23
It already already contains heaps of geospatial software. We have 12,000 github stars 10,000 forts Usually we have like 5,000 open PR Like in one month we can merge 5,000 of them open like 1,000 more easily
10:44
so this is the graph showing amount of packages and their state of Like their freshness. So this is amount of packages. This is freshness of those packages. So here are Linux distributions Debian and Ubuntu and everything else here is nix. So nix has dead more
11:08
Amount of packages which is only comparable to arch Linux I think it's even more currently but like art art. This is not official art repository, this is art user repository which contains like everything and
11:24
Our packages are more up-to-date even that Even comparing to art user repository So number of contributors is really growing each year as you can see we have now like around
11:40
1,000 contributors the problems Are always present currently we can build QGs on Mac because there is like not enough Make Knowledge I guess so another Component is nix operating system another very unique
12:02
operating system It is declaratively configured which means like you say what you want not how to get there I won't progress well, and it will give you a postgres quell It is configured using a module system software comes from nix packages to releases per year and dozens of another
12:21
very unique Features this is example of configuring nix OS so like Here you can see I said I want service open SSH Running on this port. I want postgres quell. I want these two databases to be present I want them to be backed up so service postgres quell backup, and I want to back up these databases
12:45
And I want to XFCE Desktop available in my computer. This is enough to to get full Operating system running Database and desktop so next community provides another heaps of other
13:02
software tools For for nix So what are the other thing about nix so I picked up a famous example from Mitchell Hashimoto Who is really like not? Some random person from internet
13:21
He's really a very smart IT person out or a vagrant packer console terraform walled nomad and heaps of other things he's saying that The next learning curve is like a sheer cliff hardest learning curve of probably in tech I ever learned and this is from person who knows really everything so it's quote from
13:42
2021 in 2022 he's saying is that He is you still using nix and the problem with nix is how quickly you lose touch with a reality or Reality of others in terms of managing dependency versions So so he forgot that there are some dependency health in another
14:02
world so Very positive quote and in 2023. He said I still stand by Nix is that number one most positively impactful technology. I've learned in recent years. So this is like really bold statement
14:23
So some a little bit of history, this is the guy who started whole thing in 2026 writing his famous PhD thesis alco dolstra Then there was another guy doing master thesis which kicked off nix OS operating system in
14:41
2008 we Got a module system powering nix OS in in 2015 there was a first nixcon alco is still like Very active in the next community. I met him one one month ago at the meetup So this is the first sentence of his master thesis
15:03
This thesis is about getting computer programs from one machine to another and having them still work when they get there. So it's still a valid point basic examples, I'm really Going to simplify that The biggest topic is reproducibility, which means the same out inputs
15:23
And as exactly same outputs bit to bit or back to back To achieve that everything is built in like very strongly isolated environment from host internet and with heaps of Configurations which guarantees that it's always ends up exactly like that
15:43
Whatever next builds it ends up in next door, which is special directory next flash Store and ends up in the subdirectory which is called store part, which is which is Named using unique cryptographic hash based on inputs of what you are building
16:05
Same inputs always end up in same hash and vice versa So this is the example of next on directory. I built G doll I build another G doll another G doll built QG. So if you see this is the same version of
16:22
Yeah, the the structure is has package name and version. This is the structure for packages We can build different other things. But this is example for packages. So we see we have The same version of G doll multiple times, which means that I build the G doll probably with different page or different build
16:42
configuration Here and I can run them. This is the content of the directory familiar directory structure of G doll nothing extremely special there and The key
17:00
Key Concept is that the software can't just run like the package just can't run by itself It always needs to run with all dependencies including basic C library and all those dependencies must be exactly those which That package is intended to use with exact hash
17:25
Which Belongs to that package and there's another Yeah, okay, so yeah, this is the example if you want to run G doll it needs all this software
17:41
down to glibc exactly with those hashes guarantees which guarantees the exact build and if one package needs Equally same dependencies as another package. They are nicely reused between those Package this is the very big big big advantage over flat back. For example, so you share what you want
18:06
What you can because it's reproducible if I built software on my machine It doesn't need to build to be built anymore. It is stored In cache. We have a local case on our machine. We have official next packages cache. We can have
18:24
another third party or By us built cache services and We can forever use once built Package or any kind of output so why reproducibility is important
18:40
We can have exactly bit to bit same environment for dev test UAT CI production environments we Run and build we can run at the same Software built once We need to run a test to it only once if it's
19:01
running against reproducible Software so for example for science it is even more visible you can You can share your environments with your experiment for others to reproduce that There's a famous article about the reproducibility crisis in nature magazine
19:21
2016 which is saying really fun, I mean which is Which is saying that 70% of researchers have tried and failed to reproduce another scientists experiments and even more have failed to reproduce Another so how to install the two installers of next currently both are
19:45
Good and some of them enable some features some of them Not by default there's an installation so how to use next for geospatial environment So this is a typical
20:04
Typical Project setup and the requirements I want to pick up my machine Initialize git repo and get project dependencies and start basically working I don't want to blow up my system and break other projects and over the time
20:20
I want to update my dependencies with configure and configuration rollback my changes customized with packages with patches and Yeah, I want to skip some other slides, so how to How to use
20:41
Sorry, okay So what is Geonics as a point of this presentation. It's my proof of project proof of concept Project which is building pet project development environments for geospatial Projects It provides the latest geospatial packages like one to GIS but for all Linux distributions and Mac and Windows
21:08
WSL it provides isolated project environments like Python virtual and but for all software It provides some functions for For running geospatial services like PostgreSQL and others it can build container images
21:26
So how to how to? Start a Unix project. There's a command Unix CLI in it, and it will give you two files One of them is a configuration file where you configure your
21:41
Project in the in a form that you create Some kind of shell environment so here I'm creating two shell environments one for packages one for services default is for packages PostgreSQL is for Services there are different function functions, which allow me to do so so
22:02
Now how to launch those Environments so everything I need to do in my root directory of my project is to say next develop and name of the of the Environment so if I do next develop default. I have all packages coming from default
22:23
Environment so I specified Gdell there, so I have a Gdell if I do next develop PostgreSQL I have Nicely running PostgreSQL running under my user in my project Directory if I want to add more stuff to my shell environment I
22:41
Just add them to extra packages parameter of the function there are other Parameters or arguments of the function As well, so this is extended. I added few packages and calls a message to my environment
23:00
If I want to develop Python there's another function make Python their shell Which turns my environment to Python environment? I can specify version I can specify my dependencies for example Fiona. I can have some non Python packages there, so if I re-enter my shell environment and say next develop default I have
23:25
Python prepared Environment so I can for example Try Fiona if it works it works. I can even use poetry to install additional dependencies, so I installed flask From PyPy and it works nicely with what comes from Nix. I can extend
23:47
PostgreSQL by just like saying I want PostgreSQL 12 and PG routing To be present I can add more And not of another environment for example some client environment Which allows me to easily to connect to my PostgreSQL, and I just made one more
24:06
mock web Environment just running Python server. I checked if I haven't done any mistake in my configuration these are Environments which I created immediately available for Darwin which is Mac and for for
24:23
Linux thank you and You see I'm running PostgreSQL. I can connect to database This is my web service if I don't want to Run all those services by by hand I can use process compose where I just list all my
24:42
My services in one file, and if I run process compose I have a nice UI which starts all services gives me a logs like Docker compose for example, there's even a way how to This is killer another killer features how to how to patch
25:02
Packages for example in this example. I try to create the patch which is Just like overwriting the version of gdal to something else I just use that page in my project Configuration like this and I just re-entered my
25:25
development environment and Next evaluated my change and rebuilt everything which was needed so it's not me knowing what needs to be rebuilt, but next exactly knows what to rebuild and I just Re-entered my shell environment and everything was done so so you see here all software is now using that
25:48
Custom gdal version revision 19.9. It's kirgis. It's Fiona. It's post GIS Doing that there's a way how to log packages to certain version how to update them
26:02
How to roll back to previous version you can do You can compare different comments using different dependencies And find if they break something And you access yeah So there are other similar tools like geo nix
26:23
There's a devent which is done by really like nix gurus and veterans And it has another heaps of features how to enable 50 more most now famous Programming languages using module system similar to nix OS you can start services
26:44
But just like saying I want to have PostgreSQL. I want to have some kind of containers There are other tools as well dev box It has this kind of configuration all of them are using nix so like last three sentences my conclusion
27:02
Really like after a couple of years using nix is that containers were really the best development environment until I discovered nix and Ansible Ansible was really the best configuration management until I discovered nix OS declarative module system configuration yeah, so if you are interested just come to me and
27:25
Thank you very much