Modern package management
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 26 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/19160 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
1
2
7
8
10
11
13
14
15
18
19
20
21
00:00
FreewareBuildingMetropolitan area networkComputer configurationDefault (computer science)Revision controlBranch (computer science)PiScripting languageComplex (psychology)Real numberSoftware maintenanceSoftware testingScalabilityFactory (trading post)Compilation albumDifferent (Kate Ryan album)Revision controlBuildingInclusion mapPhysical systemStability theorySinc functionNetwork topologySoftware bugDefault (computer science)Compilation albumArmPatch (Unix)Branch (computer science)Stress (mechanics)ResultantSoftware testingScripting languageMultiplication signTunisOperating systemData managementRight angleProduct (business)System administratorData centerFunction (mathematics)Computer configurationCodeSoftware maintenanceSource codeSequenceArithmetic meanSoftware developerBitCuboidAuthorizationIntegrated development environmentAdditionGradientElectronic visual displayWave packetFood energyPoint (geometry)CommutatorFactory (trading post)Web pageFreewareRepository (publishing)Water vaporVarianceFormal verificationConstraint (mathematics)Line (geometry)DemoscenePulse (signal processing)Condition numberScalabilityPersonal digital assistantSerial portState of matterComputer animation
08:28
Disk read-and-write headPatch (Unix)BuildingSpecial unitary groupElectronic mailing listNetwork topologySet (mathematics)Repository (publishing)SummierbarkeitKey (cryptography)Software repositoryElectronic signatureData typeMIDIWebsiteMetropolitan area networkHill differential equationProxy serverRow (database)Library catalogClient (computing)Repository (publishing)Revision controlGoodness of fitAttribute grammarBoolean algebraLibrary (computing)Type theoryMetropolitan area networkPhysical systemConfiguration spaceCASE <Informatik>Coma BerenicesLine (geometry)Different (Kate Ryan album)Direct numerical simulationDisk read-and-write headInvariant (mathematics)CuboidSemiconductor memoryLink (knot theory)Sign (mathematics)Sound effectComputer configurationCompilation albumBuildingPatch (Unix)Form (programming)Public key certificateNetwork topologyData managementDefault (computer science)WebsiteINTEGRALComputer architectureSystem callElectronic mailing listEuler anglesMultiplication signCondition numberGender1 (number)DampingComputer fileCombinational logicRollback (data management)Core dumpKey (cryptography)Message passingMultiplication2 (number)Set (mathematics)Source codeMereologyIntegrated development environmentRight angleComplex (psychology)Virtual machineLattice (group)Computer animation
16:57
Communications protocolScripting languageDisk read-and-write headKernel (computing)Installation artFluid staticsFreewareRepository (publishing)MultiplicationLibrary catalogComputer fontComputer configurationComputer fileDefault (computer science)SatelliteConfiguration spaceFile Transfer ProtocolMultiplication signHeegaard splittingRevision controlMechanism designFluid staticsDifferent (Kate Ryan album)Physical systemTouchscreenDisk read-and-write headBootingView (database)Scripting languageSet (mathematics)Branch (computer science)Office suitePoint (geometry)Traffic reportingExecution unitStability theoryWordWebsiteDirectory serviceGradientGroup actionSound effectServer (computing)Web 2.0Key (cryptography)Process (computing)Direction (geometry)UsabilityRight angleRepository (publishing)MeasurementEvent horizonCASE <Informatik>Library (computing)Gastropod shellElectronic mailing list1 (number)Touch typingBuildingInstallation artCommunications protocolJust-in-Time-CompilerTunisComputer animation
25:25
System on a chipMultiplicationRepository (publishing)Communications protocolLibrary catalogPlug-in (computing)SummierbarkeitConfiguration spaceMetropolitan area networkMaxima and minimaScripting languagePattern languageComputer fileDifferent (Kate Ryan album)InformationServer (computing)Multiplication signDefault (computer science)HoaxAdditionGradientLink (knot theory)Library (computing)Poisson-KlammerRevision controlRule of inferenceProjective planePersonal digital assistantLinear regressionAdaptive behaviorForestSingle-precision floating-point formatTelecommunicationSoftware developerProgrammschleifeMessage passingMassComputer configurationConsistencyDialectState of matterNetwork topologyPoint (geometry)MereologyDistanceElectronic mailing listWritingKey (cryptography)Sound effectPhysical systemWordCodeArithmetic progressionConfiguration spacePlug-in (computing)Repository (publishing)Form (programming)Pattern languageBoolean algebraCommunications protocolSource codeNP-hardComputer fileParsingDepth-first searchAxiom of choiceInstallation artHash functionSet (mathematics)Table (information)BuildingInterface (computing)Limit (category theory)Rollback (data management)FreewareDifferenz <Mathematik>Web 2.0Function (mathematics)File systemLibrary catalogDatabaseMetadataMultiplicationComputer animation
33:53
Repository (publishing)MultiplicationPattern languageScripting languageSystem on a chipMetropolitan area networkRaw image formatMetropolitan area networkPlanningPhysical systemMultiplication signGroup actionWordStudent's t-testWater vaporProduct (business)RoutingInformation securityExecution unitElectronic mailing listForm (programming)MereologyScripting languageRevision controlMechanism designSatelliteDefault (computer science)NumberAuthorizationOrder (biology)Regular graphRepository (publishing)Configuration spaceComputer fileBitCodeNetwork topologyUniverse (mathematics)Coefficient of determinationLevel (video gaming)Web applicationRow (database)Computer configurationProjective planeCASE <Informatik>Descriptive statisticsLine (geometry)Integrated development environmentInformationSynchronizationInteractive televisionDifferent (Kate Ryan album)Point (geometry)Installation artUsabilityArmWechselseitige Information2 (number)Pattern languagePlug-in (computing)Computer animation
42:22
Software testingState diagramDew pointCountingBridging (networking)Installation artSpecial unitary groupIntrusion detection systemBinary fileBeer stein3 (number)Metropolitan area networkAmsterdam Ordnance DatumInterior (topology)Computer-assisted translationExecution unitMotif (narrative)Menu (computing)Volume (thermodynamics)Virtual machineRaw image formatDirectory serviceKernel (computing)Row (database)Configuration spaceCASE <Informatik>FamilyProcess (computing)Computer animationSource codeJSON
43:47
Intrusion detection systemSoftware testingInheritance (object-oriented programming)Computer-assisted translationMeta elementSummierbarkeitBlock (periodic table)BackupInstallation artAmsterdam Ordnance DatumCountingRule of inferenceCylinder (geometry)Ultraviolet photoelectron spectroscopyArmWebsiteMetropolitan area networkDew pointGroup actionVertex (graph theory)Beer steinFreewareBinary fileComputer fileDirectory serviceMIDIInterior (topology)Demo (music)Bus (computing)Parameter (computer programming)MUDInstallable File SystemDirectory serviceLimit (category theory)GodSource codeJSON
44:39
Parameter (computer programming)BackupInheritance (object-oriented programming)Computer fileDirectory serviceBlock (periodic table)Group actionCylinder (geometry)Interior (topology)Computer-assisted translationDemo (music)MIDIBinary fileSoftware testingUniform resource nameDew pointVertex (graph theory)Server (computing)Physical systemWeb 2.0Integrated development environmentDatabaseImage resolutionSet (mathematics)Theory of relativitySeries (mathematics)Connectivity (graph theory)Message passingWeb pageCASE <Informatik>Source code
45:21
MIDISummierbarkeitBackupDirectory serviceInheritance (object-oriented programming)Block (periodic table)Group actionCylinder (geometry)MUDParameter (computer programming)Library catalogComputer fileAsynchronous Transfer ModeSystem callRepository (publishing)Address spaceSpacetimeUniform resource nameSoftware repositoryBinary fileOpen setMaxima and minimaData integrityInstallation artDuality (mathematics)Physical systemMaizeTotal S.A.Duplex (telecommunications)Service (economics)Computer-assisted translationDemo (music)Interior (topology)Software testingDew pointDivision (mathematics)RootVarianceUser profileBootingHypermediaLibrary (computing)Default (computer science)MiniDiscCore dumpLoginCorrelation and dependenceEmailComputer networkMountain passMetropolitan area networkFrequencyMobile appInformation securitySkewnessFirewall (computing)CAN busExtension (kinesiology)Gastropod shellLocal GroupInstallable File SystemMeta elementWeightReliefInformationSimultaneous localization and mappingFirst-order logicArithmetic meanLevel (video gaming)InfinityPointer (computer programming)Spherical capLetterpress printingComa BerenicesFreewareTime domainRegulärer Ausdruck <Textverarbeitung>WebsiteRow (database)Kernel (computing)Maß <Mathematik>Time zoneEmulationMalwareIntrusion detection systemBridging (networking)Scripting languageFeedbackError messageNumberFunction (mathematics)Kernel (computing)Physical systemLine (geometry)Revision controlBus (computing)Configuration spaceGroup actionMultiplication signComputer iconObject (grammar)Metropolitan area networkArithmetic progressionSource codeJSON
47:38
Computer-assisted translationDemo (music)Software testingMIDIBinary fileComputer fileDirectory serviceSpacetimeDew pointInstallation artData integritySummierbarkeitInterior (topology)BootingRevision controlKernel (computing)WebsiteComputer configurationConfiguration spaceMultiplicationConditional-access moduleState diagramStructural loadDebuggerGeneric programmingFreewareElectric currentBitMereologyCore dumpRow (database)Entropie <Informationstheorie>Installable File SystemSystem programmingBlock (periodic table)FrequencyEvent horizonError messageBridging (networking)Bus (computing)Computer networkGateway (telecommunications)Ring (mathematics)Metric systemManufacturing execution systemLibrary (computing)WeightOpen setQuantum stateSoftware engineeringNewton's law of universal gravitationPhysical systemSatelliteLogic gateCompact spaceAmsterdam Ordnance DatumPhysical systemKernel (computing)Multiplication signMereologyVideo game consoleSet (mathematics)Pulse (signal processing)Figurate numberSource codeComputer animation
48:45
WeightComa BerenicesSelf-organizationDifferenz <Mathematik>PredictabilityBuildingRepository (publishing)Multiplication signRepresentation (politics)Boolean algebraRight angleGene clusterComputer animation
50:03
Physical systemCASE <Informatik>Mechanism designMultiplication signSampling (statistics)Computer animation
Transcript: English(auto-generated)
00:00
for attending. I'm Baptiste Darussin, I'm a port committer in FreeBSD since for almost three, a bit more than three years now. I'm a source committer for two years. I'm the author and the main developer of Packaging which is a new package system for FreeBSD. I gave a talk last year to explain you what is Packaging, so this year
00:23
it's not an explanation of what is Packaging, but what you can do is Packaging and how you can do modern package management with Packaging right now. So first, I'll give an introduction on some prerequisites you need to know before starting
00:43
building Packaging, using them and deploying them. Then I'll explain how you can create your own Packages easily, run your own repositories, how you can just publish your Packages if it's for system administration because you have a huge data center with FreeBSD everywhere, and you want to publish all your Packages on your boxes, or
01:03
if you are a vendor and you want to publish just one, two, or three specialized Packages and be able to provide them to the users. I'll explain you what are the new features you have in the recent Packaging which will help you to deploy your Packages.
01:21
We have started working on Packaging base so that you can use Packaging G to install your kernel, install your system of Gradle and do things with all the base system. I'll finish with some overview of the main features on the upcoming package 1.1,
01:42
what we have done in it, what are the new things, and what we are already planning for the next versions, and yeah, if I still have some time, I will show you some examples of what we can do with Packaging G, Gels, and Beehive.
02:04
So first, why do you want to build a package instead of using the mirrors that can be provided by PCBSD people, by FreeBSD people in a couple of months, a couple of weeks? Because you may need some specialized package with non-default options. A lot of people are willing to
02:24
I don't know, remove X11 everywhere or add some support for special things. You probably want to have some specialized version and not the default provided by the port tree, for example, you want to get to keep the old version of Perl or you want to have a special version of MySQL or
02:41
anything else. You want to have your custom packages, I mean, you have something which is not in the port tree, it may be just because it doesn't exist yet in the port tree or because it's your own code and you want to package it and provide it the same way as you do with the rest of your packages, so you probably want to have your own port and build them.
03:04
And you want to manage your own stable branch of the port tree, you have an appliance somewhere, you have built all your packages based on a specific revision of the port tree, you want to keep your packages better than that, but probably you want to add some patches on it, so you want to keep it.
03:24
In the old time, what you had to be able to build your own packages was try to install pointy hat on your own system, which you might end up failing at because it's really complex setup, it's really hard to debug if you have some
03:43
weird output somewhere or some failing packages, and it has some design bug leading to not really clean packages from time to time. You may have wanted to test tinderbox, well, the problem is it's slow. I mean, it builds packages one by one and
04:05
you don't want to wait for two days or a week to be able to get your packages for your systems. It's rather complex to install, not that much, but still rather complex, and it has the same design bug as pointy hat.
04:22
So, you might end up trying to build something with Clang because it has detected that Clang is on the host that you are building for FreeBSD 8, and you don't have Clang on it, and you have some package failing because of those design bugs. And you could have written your own, your homemade scripts, a lot of people have done that.
04:42
Problem is, most of the time, the homemade script doesn't build packages in a clean room, meaning that you can leak some dependencies you will, you don't want to have in your packages. Most of the time, it's slow, it's sequential building, so one after one after one, and you get long time
05:01
before you get your packages, and you need to maintain it on your own, and so you need to follow what the new things are in the pod3 to make sure that your scripts are still compatible with the recent versions. What we have now is we have a new tool called Pudrier,
05:21
which means product egg, and it's a package building factory. It's written to do some package testing, to be able to build, to do some quality insurance on them. It can be very strict, way more stricter than Tinderbox can be, so it can detect a lot of failures you have. I won't insist on
05:42
testing and quality insurance when building the packages, but if you're a maintainer of some ports, or if you're a port scometer, you might be really interested in testing your package in your ports inside Pudrier, you will have funnier results sometimes, because it's really strict.
06:04
It's also an operating system stressing tool. It is doing so much thing at the same time, randomly, that you end up having everything on your operating system stressed. Trust me,
06:21
Dragonfly people, for example, have started using it and discovered a lot of, lot of bugs in the system, and they improved a lot their scalability, stability, and performance by just testing Pudrier on the system, because it was parallelizing so much thing different in the same time.
06:40
Anyway, so if your system survives a full build with Pudrier, you can say you have some stable operating system. So, how Pudrier works? First, it built everything in the cleaning room, in the sandbox. So, when you are building a package, there is only the thing that has been specified as
07:03
dependencies that are on the system while building. It allows cross compilation, so you can, you're on an MD64 box, you can build packages for 32 bytes, for any version of FreeBSD, as long as your, your host has
07:21
corrects its code, so usually your host is the most recent FreeBSD version available, and you can build for all the previous version of FreeBSD. We are working right now to include cross compilation on MIPS and ARM
07:42
destination, based on Stacey's work, and we currently are able to build almost half of the post tree for MIPS. It's a bit slow yet, but it will improve soon. So, not in the next release of Pudrier, but in the release after, you will have something that allow you to build the packages for embedded environments.
08:05
It's massively parallelized. If you have about, if you're trying to build the packages in the box with 24 cores, by default, it will, it will build 24 packages at the same time. You can do some tuning on that, you can say, oh, I just want to have 12, 12, 12
08:22
jails building packages and 12 threads, 2 threads per jail. You can do all the tuning you want, but by default it's all the, all the core I use all the time. It has deep integration with ZFS, which, what allows Pudrier to be very fast is because we massively use
08:40
the clone, the snapshot rollbacks from ZFS to be able to get back to clean room, but we also now are able to use Pudrier without ZFS. So, you can run it on your UFS file system, it's a bit slower, because you have to copy everything all the time, or you can use it in full TMPFS.
09:00
So, if you have a box with a large amount of RAM, you can build everything inside memory, and it's very fast. On the box we have right now available for port manager, we are able to build on a current with witness and invariant the whole port tree in 30 hours.
09:21
It supports incremental builds, that means, if you just update a bunch of ports in your port tree, and you want to rebuild them, it will make sure that only those are rebuilding, and only are rebuilt, and only it will make sure that what depends on those new ports are also rebuilt, so that you are sure that your repository is always consistent, and is always working with all pieces together.
09:45
If you want to maintain your external port tree, like you can have with EXO-DODL, or Gnome, or you can have on your own system, you can easily use PortShaker with that, there is, in PortShaker, support for Pudrier,
10:02
so, you can just set up a special port tree, which will take the FreeBSD port tree, your own ports on your own SCM, and then merge them, and you have a port tree where you can build everything. Pudrier is able to package ports, and soon, I have all the patches for that, it is also able to, it will also be able to
10:24
package base. So, you will just have to say Pudrier, package base for this version, and you'll get a bunch of packages which correspond to your own multi-base system and things. So, how, Pudrier, we have tried to get something really simple.
10:44
All the builders we used to have before Pudrier, I found them too much complex to use, and I wanted just something straightforward. So, how Pudrier works is just to say, I want a jail based on the lattice head, so I will create it, just like Pudrier, jail, dash c, dash, a name, which
11:06
shouldn't be a, it should be a g, a j, sorry, and a version, so I want head, and I want to get the sources out of SVN. It will take everything, it will build everything, and it will create for you a jail ready to build the packages
11:20
based on the lattice head. If you want to test some patches, so you're you're customizing your, the base system for your appliance, or you're a FreeBSD developer, but trying to do some modification on base, which could perhaps have some impacts on the ports tree,
11:42
you can just specify dash p, and with the dash p, you can just give your, your patch to it, it will build, it will create a jail with the most recent head, your patch applying on it, and then you can build all the ports on there, so you can do your own ex-prung. You can create out of,
12:04
you can create out of a release, so it's not the right line, but if here you, you specify nothing, it will take the sets out of our own FTP, so you don't have to build anything, so in 20 seconds you have a jail ready, without having to build anything.
12:20
Here you just specify, in that case, normally it should be 9.1 dash release, and it's done. To be able to create jails for cross-compilation, for example for MIX64, you just have to specify the architecture, MIX64, and then it will do the cross-compilation, prepare everything so that you have a jail ready to build MIX64 packages.
12:46
Now you have all the jails ready to be able to build the packages, you need a ports tree, because all the definitions to build packages are in the ports tree, so you just create the ports with dash c. You can, if you had dash p, a name, you can have different ports tree at the same time with different name,
13:03
and you can switch building from one ports tree to another, and you can play with that. If you want to create a single package, you just specify it on the command line on the berg, so you say, I want to build on this jail, I want to build apache, and you will get a few minutes later apache with all the dependencies being built from scratch.
13:24
Of course you can specify some customizations through make.conf, you can specify make.conf per jail, per combinator jail, if I use this jail, this port, this is this make.conf, and you can do special assets with the dash d,
13:42
so in that case you can still reuse the same ports tree, the same jail, but have some special options just for this particular set, so you don't have to maintain current jail based on the head, if you just have different machines and you have one jail but different sets.
14:07
And if you want to build the whole port tree, it's just berg dash a, nothing more, fairly easy. Now that you have managed to build all your packages, you get a repository,
14:20
if you have built them inside, inside Pudohier, then this part is done automatically by Pudohier, this is basically creating a catalog of all your packages so that your clients can be aware of what is available on the repository you're proposing. If you want to sign the repository, you just specify at the end of the command,
14:45
a certificate and then it will sign the repository, so the clients can make sure that the packages are the ones coming from the right side, the right place. With package 1.1, we have activated the full multi repository support, so you can have five, six different repositories where your packages are coming,
15:04
and if you want to do a simple configuration of your repository without having by hand to edit every single package.conf or saying to a user please add this line on your configuration, you can just create a simple package which will have just a small file,
15:23
a .conf which is a yaml and which is expecting a package sheet which is a URL where you can find the packages, ABI is automatically expanded on the system so it will only get the packages corresponding to the right version of FreeBSD it's using.
15:41
You can specify your mirror type per repository, so we support two kind of mirroring which are HTTP, basically package.ng just does the first HTTP request, it gets a full list of different available mirrors and it will try to get the package through the first one, if there is no package in that, it will fall back to the next one and so on.
16:03
Or you can have mirror type which is R3, basically just querying the DNS records, the SIV records and get a list of hosts where the packages should be available and it will do the same if it goes through the first one, next one, next one
16:20
until it finds the package. Sorry? Yeah, sure. You have, in the package.conf you have, you can in 1.0 specify just HTTP proxy and then it's done automatically and in the package 1.1 what we have done is now you have a key which is AMV
16:41
and you can specify every environment variable that are used by any of the libraries we are using like libfetch. Then in this repository configuration you can add a pass to a key so if you had this key in your package with this file you can just, the user may be able to have it
17:03
and you can enable or disable the repository by default. Once you get this into a package you can just say to a user do package add this pass and you'll get my configuration from my repository and your system is automatically configured to use the new repository.
17:24
So, what I ask the user is to go to my company, there is this TXT file, add it and your system is now able to get packages through my company. What protocol are supported to be able to deploy the package
17:42
we support HTTP, HTTPS, FTP, all of this comes from libfetch so we have nothing in it. In package 1.1 we added SSH support. A lot of people were complaining on package 1.0 saying that oh, it's cool, I have my packages, I have my repository but I have to maintain a web server somewhere to deploy them.
18:04
So you don't have to do that anymore. You can just share your, some keys between the host and you say you had SSH something into your package site it will just do the right thing. The only thing needed is, you need on the hosting server
18:23
you need package install and the 1.1 version. So, packaging base. Basically what I did is adding two new subcommons to put here so it's set forward for anyone you just say I want to have a package based on the latest SVN revision on head
18:47
and you can specify a script directory. The script directory is a directory where there is some YAML so basically you have kernel.script, base.script, etc.
19:02
And if you want to do some special tuning in post-installation, post-upgrade post-installation, everything when you are setting up your JIT. For example, if you want to add it on my set I have something because I know exactly how the FS tab will be after an installation
19:20
so I just push this into the post-install script if there is no FS tab yet, create it with this line. You can create it out of the... if you do that, it just fetch a lot of snapshots that are available on the FreeBSD FTP server and then it creates the packages out of those snapshots
19:41
that you don't have to build. I can only test it with head and stable but it should work the same with releases and with the different releases branches. So, the good thing with it is you end up with this list of packages. I haven't been so far into splitting base
20:02
because I don't want to go into a bagged shell about that all I wanted is to be able to show that yes, we can do that with package-ng if people are willing to go in that direction or willing to use this, we can do this. It's very easy to split it in more packages than this
20:21
if you want, for example if you want libc++ being used on your system you can have a package which is libc++ and it will replace the libs on our C++ on the system. You can imagine everything you want in that case.
20:40
If you're building on head and on stable as I don't have any version, what I do is I give you the major version and then I had the date where the snapshots are from. What I've been able to do so far by package-ng base is
21:00
before that, when you had to upgrade from 8.3 to 9.1 you use FreeBSD update it was really, really, really, really slow, long it works, it works fairly well and so you had to first upgrade your system reboot, install all the new world
21:21
and upgrade your packages and because you are changing the ABI you use the static version of package and you don't use the normal version. Upgrade-f say, ok, all my packages I have on my system please reinstall them from the latest available in the remote repository and because the ABI is automatically expanded
21:42
package-static will detect that it doesn't need anymore to fetch packages from the 8.3 repository but it will take the one from the 9.1 repository automatically you don't have to modify any setup for that it's totally automatic. What I have been already able to do
22:02
and what you could do once I release next version of Pudria is you can just do package install-f when you do dash-f it means that if it's already installed you force reinstallation or upgrades of those files. First you need to do or it's not world, it's base, but anyone
22:22
first you need to do those two ones or if you want to be really clean you should first do pure-null, reboot, then world but most of the time you can do those two ones but not the packages that come with them because otherwise all the script from the packages will almost fail you will upgrade packages based on the ABI you have before having upgraded
22:45
so first upgrade based on the world then reboot and you can just upgrade-f the same way. Yes? What we do is currently we don't do anything
23:01
so what I do is on my scripts I just remove from on my post-upgrade script when I was creating the script here I just know exactly I have a couple of files that are not the official one so I remove them from the set
23:21
and I have in post-upgrade something that tests okay, this one, if yes I create, if yes, I just, I don't touch it if not, I upgrade it so I do it manually currently this way thing is, in package-ng we have a mechanism to mark a special file
23:42
as being a configuration file problem is, when I coded this I was only thinking about packages which have only one, probably two configuration files when I tried to use this with Base it, I end up with
24:01
slash etc being totally messed up there was files everywhere and I had to go through each one and say is there a difference? so it was really really not a good idea the way I did it so I'm reworking it, it will be in 1.2 and basically what I'll do in 1.2 is if a file is marked as a configuration file
24:21
then before grading I'll check if the file has been modified since the previous installation if yes, I keep the old one I create a package new one belonging to this one and if I'm able to automatically merge I will merge them automatically
24:41
not by default but you will have an option in package-ng try to automatically merge if you can so normally after an upgrade you will end up with just a couple of files with package new so it will be fairly easy to create your own merge master and if you activated the auto-merge most of the time it will be just straightforward
25:08
so, okay so, what we have done in package 1, once in package 1 is very important for us we have stabilized a lot
25:22
the foundation of what we have written for us so, we have been able to activate by default full multi repository support then you don't have any more single repository support everything is full multi repository by default we have added something which is
25:40
an incremental catalog update because so far what we did each time you did package update to get the latest catalog from the different repository is take the full metadata bring them so you have when you have 24,000 packages you have a huge file not that huge but still too huge for me to download so now what we do is we first fetch a small digest
26:02
and the digest we have a look at the database you already have on your system and if the diff between what is remotely and what is local is small, then we just fetch the between and we upgrade it we upgrade your installation if the diff is huge then we took the raw one and replace yours
26:23
when we release 1.0 I always say that the library API wasn't stable because we were pretty sure that a lot of people were coming and saying oh, why don't you do that that way why don't you do this that way I don't consider myself as a strong C developer so I was expecting a lot of people to come and say
26:42
here is the right way to do a library we didn't had anyone saying such a thing since 1.0 but still we have simplified the API a lot so now I consider almost stable with 1.1 it will be restable stable with 1.2 but you can start using it in your own project
27:01
with 1.1 there won't be there won't be huge differences with the next version it will be restable with 1.2 which means we won't break any API any API until 2.0 we have done some massive performance improvements every single place
27:21
where we had a lot of loops everywhere now we use hash tables which are far which are which are very faster than when we used to have we have added a couple of new commands a lot of people were willing to I want to upgrade everything, ok, I understand the fact that my set should be consistent so I upgrade my whole system
27:42
at all but for this particular package I want to force it to keep to stay at the old version so now you can just package lock the package and the package won't be upgraded anymore won't be detached anymore if you really want it to be upgraded you first need to unlock it a lot of vendors
28:01
have been looking to package.ng recently and asking to us I want this in the package, I want this information in the package and we have two choices either hard code every single new information in package.ng or provide them some free form annotation so now you can just add your own
28:21
specialized things into package.ng with package annotate, package annotate will give you a list of key values so you can say I don't know, build from this server or you can say this is copyright this company I don't know I don't know what you will want to add
28:41
to this, but you will be able to add all the information you need and it will be bundled into the package so the user will keep it so I already told about the SSH protocol and we now have a new plugin interface so the plugin interface has a public API, so you don't have to build the plugin inside
29:01
the package source code you know, you just have to include package-plugin.h and you build, you link to the library, we provide a package config thing so that it's very easy to write plugins why we did that is a lot of people want
29:21
features we don't want for FreeBSD, but we want them to be able to have them we want to keep package-ng simple the more simple we can because we want to be able to review it all the time, we want to be able to understand every single part of the code each time but we also want people to have their own comments
29:41
and we also want to have people to to have their own hooks inside the instrumentation of the installation of the upgrade or everything like this so we have written a very simple API, actually I think it's really good
30:00
I'm not planning to change it anymore what does the API basically you have to initialize the plugin when you initialize the plugin you just specify that I would need that kind of configuration, so I want a key value named this I want a boolean named this but you don't have to do the parsing of the
30:22
configuration file, it's automatic so you will have a configuration file which is consistent with package.conf, it will be a YAML file you don't have to do all the parsing all the checking if the values are there or not you just say in the initialization here is my list of configuration I need and you get them
30:42
so later you just have to check it's fairly easy we have two kinds of plugins you can have new commands each time you have a plugin with a new command if you do package with nothing you will have the list of commands and all the new commands are in if you have already used Mercurial it's been the same than when you had plugins
31:02
to Mercurial, so you just have the list of new commands and you have the hooks the hooks can be can be done in almost all places of of the instrumentation of the upgrade the installation you can do each time I'm
31:23
starting upgrading a package I execute this, each time I'm starting the installing a package I execute this each time someone is searching something, I execute this, before this after, etc we will be proposing two different examples for this
31:40
one which is a new command which is package plugin serve package plugin serve will basically just do package serve package serve and pass to your repository and it will embed a web server on it so you can just choose it well, it's not useful anymore now that we have SSH, but still
32:02
someone can want it and a plugin that I think a lot of people are working for which is package plugin dash dfs package plugin dash dfs is expecting a configuration which is the list of the dfs file system you want to snapshot before every single package manipulation
32:23
and you have a pattern on it if I remember well, you can have a pattern saying I want the snapshot be named like this, this, this, and this so each time you will do package upgrade, you can snapshot first your system so if something is wrong, you can do my hand, the dfs rollback
32:40
you can imagine all kind of plugins, I know that some people are already working on what I heard was a package build subcommand, which will basically be port master but merge inside package plugin there was
33:01
a package tree which just output all the install packages as trees so that you know exactly where things comes from and where they are going there was there was a company which was adding some plugins to be able to
33:20
activate they had some active passive versions of some packages so through a special plugin they were doing package activate this version, package deactivate this version so limitation is when you use package static, you are not able to use plugins anymore
33:41
so before going to the next version one of the other things we have in package 1 is now all the options are self-documented, if you do package dash v, you have the list you have the version of package g if you do package dash v v you have the list of options
34:01
you have and what is their current setup and if you add a new v, then you have a small description of what each option is so normally we try to get the package.com files up to date but if sometimes we miss something, you'll have it
34:21
anyway in package dash v v and after dash v v v, you have the list of repositories which are activated for your system so for next version of package-ng we are planning some things so currently we have two summer of code projects three, exactly
34:41
one which will happen for sure because I'll mentor this one which is providing us a pluggable solver there is a lot of universities, laboratories which are doing researches based on solvers I've been contacting my patrons saying, oh, I want to
35:01
test all my new algorithm, I want to push some students working on SAT solving and I want to do this on package-ng so it's nice but I don't want them to break the working solver so what will end up is providing a default solver which is the one we have now in package 1.1
35:22
which is different from the previous one and we will allow anyone to write a plugin with its own solver and if one day someone came with a solver that is way better than the one we have, we can just switch it push inside package-ng as default the one that works and still allow people to work on this
35:42
this same summer of code the students will also work on providing all the infrastructure to get the requires provides features, so instead of saying I need, because I have a script in my package, I need this version of Perl I will just say I need Perl, I don't mind
36:01
whatever version you use so this is basic for most of the package system, but on FreeBSD you are you have to depend on the special version of Perl for this, which is not good or if you're installing a web application I don't want to say I need apache2.2.4 I just want to say I need an
36:21
HTTP server, whatever it is so we will provide all the foundation in during the summer of code to be able to do that in package-ng and then we'll have to work on the port 3 to be able to provide the same feature which would be hard, but we will work on it, yes?
36:41
there is a way to say that inside package-ng, but there is no way to say that from the port 3 so we are bounded by the port 3, we want to bring a lot of new things, but the port 3 is keeping us far from those kind of things which should be the way it should work
37:01
and the old thing that is preventing us to go really further in package is we still have to support the old package installed, so the port 3 can't be moved to all those new things because of package install that's a huge problem anyway, we also in the summer of code want
37:21
to improve the mutual repository support currently, mutual repository support is you have all this list of repository we're going through them, we're solving saying ok, I'll take this package from this one this package from this one but you probably want to say I want this package to only come from this repository so even if it's updated on another on this one, I know it has my option
37:41
it is likely working, I have won't currently if you want to do that you just have to lock it unlock it after you've got everything and say package install dash r the name of my repository install this one this is not very user friendly
38:01
for this case, so we want to be able to say this one is stuck to this repository or we want to say this repository has a higher priority over the others currently the priority is the order you have defined the files, the files I read so, on your configuration file if you make numbers
38:21
before the name of your repository then you get the name of your configuration file and it will be read in the right order and in the way you have priorities on the repository, so if you have custom repositories, it's custom options and fall back to the freebies you want for all the rest, push yours first and the freebies you want as a second
38:44
yes, yeah, that's no, there is two
39:01
there is a priority on the repository and the fact that you can mark this package to be preferred on this repository and only this repository both will be done during summer of code after that, what we plan is to provide a mechanism to skip some files when you're installing on your system if you're in an embedded environment, you don't want
39:22
the docs you don't want, the man page you don't want I don't know what you probably have some special things you don't want all the .a files you don't want, I don't know so there is multiple ways to do that, you can do that by building the package without those files or you can build the package as a full sync
39:41
but when you install, you just skip installing some parts so you'll have some globing pattern a regular expression, you will be able to add into your package.com to say I don't want this we want to eliminate as much as possible, but I really want to have no script one day
40:01
at all, on any packages Posting scripts are great because I want to do some actions after an upgrade after the installation, but the problem is most of the time they are not reviewed by security people so you can do strange things second thing is
40:20
if you do cross installation and you're installing in an nd64 box, you're installing in an ARM root, for example then it will try to execute things with the ARM if you are not able to run because you're still on nd64 it's something we really don't want in the future
40:41
to keep any scripts in there what we want to do is to provide a list of, a bit like what does cfngine which is providing a list of actions you can do those actions are coded inside package-ng saying add a line to a file you don't do echo or something
41:02
whatever you want, you say add this line to this file at this point, or remove this line or this file at this point we want to be able to provide this list a list of default actions yes
41:26
you could do the cross installation with a QEMU for sure with the emulation, you can do it, but yeah, there is a security reason and, anyway, it's slow, you know, it's not straightforward
41:41
you need to have QEMU installed you can do cross installation just out of a simple make-release, for example which is what I will want to do one day yes? yeah, we have talked
42:01
about that, yeah, but still I find it ugly to have package with scripts yeah, we have thought about that, so we will one day end up with that, yeah so, now I can just I'll try, I hope it will work
42:22
but I'll show you a simple example of using using packages for base you can see, I don't know where, that one
42:48
that screen, top row ok, that works ok, so
43:00
the use case is I want to provide some Beehive virtual machines but I first want to be able to work on setting up to setting them up before starting the virtual machine, so I probably want to first create a jail, configure everything in jail, because jail is very simple to use and then, if I'm happy
43:20
with my jail, I can just add a kernel into the machine and then say, ok, now it's VM for Beehive, so what I do is I create an empty directory I have already prepared ZFS volume to be the raw device for this, so what I do
43:42
is just formatting this, so I'm sure that there is nothing in it the directory is still, I need to mount oh yes
44:10
ok, so I mount on it, I have nothing in that directory so, I create
44:20
a persistent jail so, I have
44:40
my VM, my jail which is started then, I already have the sets created on my system and I have a web server running so, what I need to do is to specify the ABI, I'm installing the package too, because we are automatically determining the ABI based on binsh, which is not there yet
45:02
if you need to specify, to do some resolution, you don't have any etc-resolve.conf you can just add an environment variable, which is name server equal 8.8.8.8 for example, and then the resolution will be done that way so, in my case, I just want to install for this database
45:25
ok, we have tried to improve the bit, the output of package engine 1.1, so now you have the number of actions that will be done
45:41
at the beginning of the lines we are really willing to get feedbacks about people, we want to improve all the error message, the warnings the output of the tools so that it's really good for users, so I still have some warnings I shouldn't have yet, but I haven't fixed all my scripts, bypassing my scripts
46:03
right now, I have my jail which is installed, so if I go inside the jail, I should be able to get in so I'm in the jail everything is there, so I can do my configuration probably install things, I have no package
46:21
inside the jail, so I can't determine what's in it but if I do I know that I have the FreeBSD bus system installed with this version inside the jail now
46:40
I will I will add I think I'm happy with this so I will add the kernel inside the jail
47:13
so now I just need to stop the jail
47:21
human and I can start beehive with this
47:40
and normally it should be booting a FreeBSD system so if I had some
48:02
I didn't get time to create new sets so I said normally ok, so I forgot to
48:21
configure the last part of I forgot to configure the console so anyway normally I should be able to log in, but I don't have the console activated, so I don't get into it but it has booted, so it works if you have new sets, you can just do package upgrade and you can upgrade both the kernel
48:41
and the system so we have just yesterday relocated the package engine repository inside the FreeBSD organization in GitHub people can know where it is if you want to have a look at Pudria right there
49:00
Pudria is having no dependency at all it's only something like 2009 of shellcode and it's readable really, it's not because it's shellcode, it's really readable so you can have a look at both do not hesitate to ask us any questions there is also Brian, who is over there has done a lot of work with something on Pudria
49:22
and package engine, if you have any questions now yes yeah
49:40
we yeah, in the delta diff between packages, you mean I just fed the delta between we are we were working on this, so first we need to solve one thing, is the predictability of the package building currently, each time you build a package you get a diff from checksums so you can't make diff
50:00
based on this so, we had some work to be able to do that, except that for packages like Python and everything, almost everything that have some bytecodes, we get into problem, because when we did that, we basically faked all the dates
50:20
to be epoch, and make sure that everything is from epoch and the Python was not liking it at all, same for Perl I don't remember, we had some problem like this, but for the base system this has already been solved by FreeBSD update, so if we use the same mechanism as FreeBSD update, which is not
50:40
yet the case on what I've done we should be able to go that way so yes, it's something which is planned it's really not easy to get to, but we really want that, because if I upgrade base or libreface which is almost as big as base, I just
51:00
want to get the new bits, I don't want to get the full package each time ok? yes? oh where was it? no? got it
51:28
thank you