We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Happy 5th anniversary pkg-provides

00:00

Formal Metadata

Title
Happy 5th anniversary pkg-provides
Title of Series
Number of Parts
542
Author
License
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
In 2018 here at Fosdem, I publicly introduce a new port called pkg-provide, a pkg plugin to perform reverse searches and found the package you must install to have a certain file. Started as a proof of concept about how to build pkg plugin, pkg-plugin is now five years old and gain hundred of users around the world. It's probably the perfect time to tell you how it all started, reveal the real numbers of the audience, and talk a little bit about the future
14
15
43
87
Thumbnail
26:29
146
Thumbnail
18:05
199
207
Thumbnail
22:17
264
278
Thumbnail
30:52
293
Thumbnail
15:53
341
Thumbnail
31:01
354
359
410
FreewarePattern languageQuery languageDatabaseComputer fileTrailMaxima and minimaSoftware developerSoftware maintenanceComputer filePlug-in (computing)Library (computing)MereologyElectronic mailing listMultiplication signData managementNumberContrast (vision)Projective planeZoom lensDatabaseProcess (computing)Server (computing)WhiteboardBitRevision controlClient (computing)Cartesian coordinate systemDensity of states1 (number)Computer architectureInternetworkingNeuroinformatikRight angleMatching (graph theory)StatisticsVideoconferencingArmData storage deviceMaxima and minimaOrder (biology)Correspondence (mathematics)EmailSynchronizationCompact spaceInternet service providerDifferent (Kate Ryan album)Euler anglesCommutatorFreewareDiagramComputer animation
StatisticsArchitectureFingerprintDatabaseBuildingSpacetimeComputer fileDisk read-and-write headData recoveryFreewareBand matrixData compressionFile formatFile formatMultiplication signDatabaseComputer fileBitGoodness of fitConnected spaceServer (computing)Computer architectureDisk read-and-write headMereologyElectronic mailing listSound effectSpacetimeSoftware developerBefehlsprozessorData storage deviceNeuroinformatikInternet service providerNumberCASE <Informatik>Uniform resource locatorStandard deviationBand matrixPublic key certificateControl flowRegular graphProcess (computing)SummierbarkeitError message1 (number)Thomas BayesStructural loadFile archiverNetwork topologyComputer animationLecture/Conference
Computer fileCore dumpDatabaseRevision controlNetzwerkverwaltungCore dumpServer (computing)Software testingRevision controlTraffic reportingLine (geometry)QuicksortSubject indexingCASE <Informatik>Binary code2 (number)Coefficient of determinationMathematicsData storage deviceMaizeDatabaseMultiplication signGame theoryMaxima and minimaComputer configurationSound effectSlosh dynamicsHypermediaComputer fileRegular graphResultantFile formatBitInformationSoftware developerFreewareGoodness of fitComputer architectureReal numberSubsetData compressionRegulärer Ausdruck <Textverarbeitung>Standard deviationFlow separationBinary fileComputer animationLecture/Conference
Lecture/ConferenceProgram flowchart
Transcript: English(auto-generated)
so uh start Yes, I put a who am I so Rodrigo Solio, so I'm free BSD poor commuter in since 2014
What else so yes, what speak it provides I don't know if how many of you knows package provides a few ones, okay? So I'm sorry Yes, so this part of this so as you know you we have package pick a G
which is a free BSD command used to manage packages and package was designed to be extended with plugins, so you can roll with your own plugins to run special commands handle hooks
replace existing commands or access the package database just for Statistics or have numbers or whatever So package provide basically is a plug-in you can you use with packages we extend package to Do a special thing is
matching a found found the packages for a specific file basically you have you you are looking for which package I need to install to have this far in my computer and Usually is you you start looking on the internet you google a little bit to know exactly what the package you need and
Because sometimes the packages doesn't provide the name of the files we have inside the command or song So basically it's it's a way to found is a reverse way to find with package
Correspond to a specific file it could be a library unexecutable or just a header like H file So this is an example, so you just start package provides and the lib MPG one two three dot is so and give you
the list of packages who Provide this sorry for the contrast. It's not very good so you have you have a For this sleep you have a two two five two packages We have the empty MPG one two three which is probably the main project who provides a file
But you see that zoom video conferencing also provide this library in another version probably For own and basically it's a Linux compact library So As you can guess package provide is a client so you have in your computer, but it's also a database
Who contains the full list of files? you can install if you start all the packages existing in FreeBSD and This is a lot for just
One release and one architectures we have one nine million files Installable, so this is a huge database you we have we have to be up so Just I put here. This is the numbers for the
latest version so the 13 and if you look at the different versions of FreeBSD and different architectures you see how many how many files are and basically this is based on the packages you can build on FreeBSD for this architecture and version, so
you you can probably see which are the the good architectures the attitudes who works and you have a lot of applications and those architectures you who are a little bit Not really use or we are missing a lot of a lot of tools
Who doesn't be out for the those architectures you have the also the the new ones like the army? 7 who appear suddenly and it has a big big number and you have the army 6 who is Declining a little bit. It is just still things still works, but
As the number shows it's It's not that that popular than arm v7 so This is basically how?
Everything works. I have here because I host the Database server myself, so I have the the package provide server here you have the clients who connects to upload the database and I use the package
FreeBSD servers to sync and have the list of files because this is The best and most efficient way to have the full list of files in a minimum of time Because of course you you can guess it's it's not always possible to build all the packages
Just to have to list this character nightmare So Basically, yes, I put some sometimes about these three so I start it on 2017 I Have multiple motivations. The first one was to stop doing grab on
Ports when I'm looking for a file is nothing more terrible than you try to compile a New a new project and say oh I'm missing these something age file and you start dripping Around and it's take hours just to have the full list of dependencies
So this this way You you you can found it easily and and have the exact number and also you can Have a full list so you do the order of the of the the package is providing the same file so you can choose
The right one who match for you. Usually it's not it's not the first one because some some Some packages bring a lot of dependencies so you can choose the the right one and my second motivation was
to write a package plug-in at this time package just Was a new thing This we had this plug-in feature and as much as you know probably the first Plug-in available for pick G and probably the one of the who works is still working
I have really real users. But yes so So yes package provide was introduced as the FOS them the when you're late when you're after in 2018 and
This is where the story start This is the the daily connections of people who access the server to upload the database Looks not that much but I consider package provide like Consider package provide a developer tool so I don't expect
regular people use it that much because it doesn't too much sense and And Also, we have only people access only when they perform our package update I don't upload the database every time so when you are doing an update
We you have updated at this time the plug-in looks if we have a new database So it downloads it it's right to be oh So, oh, yes Yes, this is what happens when you all your own server from time to time you have a breakages and
in this case was some kind of certificate issues The some some numbers the question is from where people come from and as you can see
It match a lot the FreeBSD developers Big so you have people mostly USA Canada some people in South America mostly in Europe, we have some people as yeah, Australia and Unfortunately, we don't have too much people born Africa
Who don't use people not like PSD there? So If you take the numbers, yes, we have the the big the big chunk of Users are in the United States and you have a Germany and France. We are
quite the same numbers And yes, and after that you have a lot of countries around the world I Sometimes yes, you have a What's funny is we we have a Russian Federation and Ukraine and both have the same number. I
Have the same number of people coming from both countries. So strange and From time to time you have some special guest so I for a couple of months I have people from St. Kitties and Nevis and food if you don't know St. Kitties is a small island here just
just close to Cuba and it's basically just a place to put money and Register your companies when you have things to hide so, okay. We have developers there
Just Yes a quick look about the database as I say the database so we have at this time we have a 58 Thousand of ports If you consider we have a three major releases and six architectures that
brings you a lot of ports, so As I say it's something you can build by your own because I don't have enough space or CPU Also to obtain the list of files
It's not possible to trust package list because those files are not fully complete sometime you have replacements we don't Do the build and I can just download as if the packages because it needs time and space to
Download the full list of packages and also have impact on mirror performances. So I found a Simple way is I try to abuse The the package file format as you know packages are just archived who contains
two two times the manifest of The the package and the second many effects which is a full manifest has the list of files Provide for the package then comes all the files And the thing is you can just use the head
HTTP request to only download the first part of the package and Let's say we took we can extract the manifest because it works and If the JSON file is valid you're wrong. You're good
If you if you if you have an error, you just increase a little bit of size Just remember it for the next time. So this way you can just download the only the manifest part of the files and the good thing is the HTTP head is a totally standard method
So we are not trying to abuse the HTTP server and we are not producing or breakages in connections We are saving people bandwidth Because we are we are not we are not using the the package mirror
Just for us and we save our also everybody's time Yes, so talking about the database itself at the beginning I Design it as a just as a full text file Just because it's easy to read and search
But it I have I have this problem. It was too large. It was 36 megabytes Even compress Okay So in 2018 I'd be busy con I talked with talking with Marcus P about the
Thing he suggests me I can use the locate file format locate is a Unix standard tool who Store all the try to sort all the files you have in your computer. So it's easy to found a place and use a
Nagorin called big ram to store the the data's and this this file format is pretty efficient because if we once once generated and compress it I reduced by two to not the size of the file, so
It's it's pretty convenient because Instead of downloading for a couple of minutes the database we can have it just in one one minute and something It's pretty similar with the package itself. So in time
So the package provide usages as I saw yes, I show you you can just perform reverse searches You're looking for a file give you the name of the package But you can also use it to identify Unexpected the files in the packages like
core dams things happen sometimes people commit and the commit comes with a binary files or dams because of this happens And also You can use it to search for duplicate files when you have two packages installed the same file and conflicting
for a long time I report this this information to to the developers and I try to track exactly how much conflicts we have and Someone decide or Stefan Stefan Ester decides to start looking at it and he used the database
The database I provide to to search for file conflicts and try to fix it To to to to avoid conflict during a package install because package the tool
Can handle conflicts, but the handle the conflicts the worst way It say okay you have a conflict so or you can't install the new package or you have to remove the old one so there's no way to and sometimes you you have under this a lot of funny things because you have a
Binary who have to say exactly the same name doing completely different things Sometime is a game and the network manager and Wow so yes, so oh yeah, and I still having six minutes, so
If you have questions, please Yeah, oh the options no I What I take is the real
files available in in package Servers so is the real thing you have if you install a package. It's not something. I'm sorry the question was if I Scan for options and the answer is no. I just try to
track what's on the package Servers and the real files installed if you install a it's not it's not use the custom If you have a custom Package server it doesn't bring you the sort of files you are installing. Yes. It's just for
Standard instant free BSD packages the way We build it Yeah Oh The first okay, how long is to how long is take to
Scan the packages and build the index along so the first the first con it's can take about I Think it's I saw I didn't do this for a long time, but I think it's like half an hour
30 minutes something like that and once Once I build it I can I can just update in a couple of seconds from one minute so it's that the the good thing is if I consider because I We have a file
Manifesto with All the packages on the version so I can see which package change so I'm concerned I just look at the package who change the version and Based on that and just recovering the Manifesto it takes yes, it takes a cup
I think the maximum is like a five minutes for for for an architecture and the version Yes The database I used to locate so it's Yes, he's a locate format
So because because it's it's really it's really convenient for this kind of usage
Yes, yes, I stored that and yeah Oh Yeah, yes, I I include because the
The way big RAM works it takes just a line and Just a line is it is a path So it's considered a path and there's no extra data around so you can you can use so I just include
The the package name inside inside the path and use a special character yes as a separator I'm in this case. I use the the star Character because I don't expect people to put it put a star in the package name
Yes, absolutely and and I use I use a regular expression for search But the the result is it's quite a bit acceptable In bytes
Compressed it's 17 exact compress it 70 megabytes. Yes Around because it depends on which version you're looking at, but it's basically yes. It's around this side Yeah
I Yes could be Could be Yeah, sorry the question. Aha. This was a trick just to see if I can repeat Yes, the question was was why not use a slash instead of star because it's like a slash made more sense
Probably yes so but I Yes, we can we can probably Oh Good question
No, I just see happy people