Happy 5th anniversary pkg-provides
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 542 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/61401 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2023373 / 542
2
5
10
14
15
16
22
24
27
29
31
36
43
48
56
63
74
78
83
87
89
95
96
99
104
106
107
117
119
121
122
125
126
128
130
132
134
135
136
141
143
146
148
152
155
157
159
161
165
166
168
170
173
176
180
181
185
191
194
196
197
198
199
206
207
209
210
211
212
216
219
220
227
228
229
231
232
233
236
250
252
256
258
260
263
264
267
271
273
275
276
278
282
286
292
293
298
299
300
302
312
316
321
322
324
339
341
342
343
344
351
352
354
355
356
357
359
369
370
372
373
376
378
379
380
382
383
387
390
394
395
401
405
406
410
411
413
415
416
421
426
430
437
438
440
441
443
444
445
446
448
449
450
451
458
464
468
472
475
476
479
481
493
494
498
499
502
509
513
516
517
520
522
524
525
531
534
535
537
538
541
00:00
FreewarePattern languageQuery languageDatabaseComputer fileTrailMaxima and minimaSoftware developerSoftware maintenanceComputer filePlug-in (computing)Library (computing)MereologyElectronic mailing listMultiplication signData managementNumberContrast (vision)Projective planeZoom lensDatabaseProcess (computing)Server (computing)WhiteboardBitRevision controlClient (computing)Cartesian coordinate systemDensity of states1 (number)Computer architectureInternetworkingNeuroinformatikRight angleMatching (graph theory)StatisticsVideoconferencingArmData storage deviceMaxima and minimaOrder (biology)Correspondence (mathematics)EmailSynchronizationCompact spaceInternet service providerDifferent (Kate Ryan album)Euler anglesCommutatorFreewareDiagramComputer animation
07:33
StatisticsArchitectureFingerprintDatabaseBuildingSpacetimeComputer fileDisk read-and-write headData recoveryFreewareBand matrixData compressionFile formatFile formatMultiplication signDatabaseComputer fileBitGoodness of fitConnected spaceServer (computing)Computer architectureDisk read-and-write headMereologyElectronic mailing listSound effectSpacetimeSoftware developerBefehlsprozessorData storage deviceNeuroinformatikInternet service providerNumberCASE <Informatik>Uniform resource locatorStandard deviationBand matrixPublic key certificateControl flowRegular graphProcess (computing)SummierbarkeitError message1 (number)Thomas BayesStructural loadFile archiverNetwork topologyComputer animationLecture/Conference
15:01
Computer fileCore dumpDatabaseRevision controlNetzwerkverwaltungCore dumpServer (computing)Software testingRevision controlTraffic reportingLine (geometry)QuicksortSubject indexingCASE <Informatik>Binary code2 (number)Coefficient of determinationMathematicsData storage deviceMaizeDatabaseMultiplication signGame theoryMaxima and minimaComputer configurationSound effectSlosh dynamicsHypermediaComputer fileRegular graphResultantFile formatBitInformationSoftware developerFreewareGoodness of fitComputer architectureReal numberSubsetData compressionRegulärer Ausdruck <Textverarbeitung>Standard deviationFlow separationBinary fileComputer animationLecture/Conference
22:28
Lecture/ConferenceProgram flowchart
Transcript: English(auto-generated)
00:06
so uh start Yes, I put a who am I so Rodrigo Solio, so I'm free BSD poor commuter in since 2014
00:24
What else so yes, what speak it provides I don't know if how many of you knows package provides a few ones, okay? So I'm sorry Yes, so this part of this so as you know you we have package pick a G
00:45
which is a free BSD command used to manage packages and package was designed to be extended with plugins, so you can roll with your own plugins to run special commands handle hooks
01:01
replace existing commands or access the package database just for Statistics or have numbers or whatever So package provide basically is a plug-in you can you use with packages we extend package to Do a special thing is
01:21
matching a found found the packages for a specific file basically you have you you are looking for which package I need to install to have this far in my computer and Usually is you you start looking on the internet you google a little bit to know exactly what the package you need and
01:45
Because sometimes the packages doesn't provide the name of the files we have inside the command or song So basically it's it's a way to found is a reverse way to find with package
02:00
Correspond to a specific file it could be a library unexecutable or just a header like H file So this is an example, so you just start package provides and the lib MPG one two three dot is so and give you
02:21
the list of packages who Provide this sorry for the contrast. It's not very good so you have you have a For this sleep you have a two two five two packages We have the empty MPG one two three which is probably the main project who provides a file
02:41
But you see that zoom video conferencing also provide this library in another version probably For own and basically it's a Linux compact library So As you can guess package provide is a client so you have in your computer, but it's also a database
03:06
Who contains the full list of files? you can install if you start all the packages existing in FreeBSD and This is a lot for just
03:21
One release and one architectures we have one nine million files Installable, so this is a huge database you we have we have to be up so Just I put here. This is the numbers for the
03:41
latest version so the 13 and if you look at the different versions of FreeBSD and different architectures you see how many how many files are and basically this is based on the packages you can build on FreeBSD for this architecture and version, so
04:03
you you can probably see which are the the good architectures the attitudes who works and you have a lot of applications and those architectures you who are a little bit Not really use or we are missing a lot of a lot of tools
04:23
Who doesn't be out for the those architectures you have the also the the new ones like the army? 7 who appear suddenly and it has a big big number and you have the army 6 who is Declining a little bit. It is just still things still works, but
04:46
As the number shows it's It's not that that popular than arm v7 so This is basically how?
05:01
Everything works. I have here because I host the Database server myself, so I have the the package provide server here you have the clients who connects to upload the database and I use the package
05:22
FreeBSD servers to sync and have the list of files because this is The best and most efficient way to have the full list of files in a minimum of time Because of course you you can guess it's it's not always possible to build all the packages
05:41
Just to have to list this character nightmare So Basically, yes, I put some sometimes about these three so I start it on 2017 I Have multiple motivations. The first one was to stop doing grab on
06:04
Ports when I'm looking for a file is nothing more terrible than you try to compile a New a new project and say oh I'm missing these something age file and you start dripping Around and it's take hours just to have the full list of dependencies
06:25
So this this way You you you can found it easily and and have the exact number and also you can Have a full list so you do the order of the of the the package is providing the same file so you can choose
06:42
The right one who match for you. Usually it's not it's not the first one because some some Some packages bring a lot of dependencies so you can choose the the right one and my second motivation was
07:01
to write a package plug-in at this time package just Was a new thing This we had this plug-in feature and as much as you know probably the first Plug-in available for pick G and probably the one of the who works is still working
07:22
I have really real users. But yes so So yes package provide was introduced as the FOS them the when you're late when you're after in 2018 and
07:40
This is where the story start This is the the daily connections of people who access the server to upload the database Looks not that much but I consider package provide like Consider package provide a developer tool so I don't expect
08:05
regular people use it that much because it doesn't too much sense and And Also, we have only people access only when they perform our package update I don't upload the database every time so when you are doing an update
08:24
We you have updated at this time the plug-in looks if we have a new database So it downloads it it's right to be oh So, oh, yes Yes, this is what happens when you all your own server from time to time you have a breakages and
08:47
in this case was some kind of certificate issues The some some numbers the question is from where people come from and as you can see
09:01
It match a lot the FreeBSD developers Big so you have people mostly USA Canada some people in South America mostly in Europe, we have some people as yeah, Australia and Unfortunately, we don't have too much people born Africa
09:23
Who don't use people not like PSD there? So If you take the numbers, yes, we have the the big the big chunk of Users are in the United States and you have a Germany and France. We are
09:40
quite the same numbers And yes, and after that you have a lot of countries around the world I Sometimes yes, you have a What's funny is we we have a Russian Federation and Ukraine and both have the same number. I
10:02
Have the same number of people coming from both countries. So strange and From time to time you have some special guest so I for a couple of months I have people from St. Kitties and Nevis and food if you don't know St. Kitties is a small island here just
10:27
just close to Cuba and it's basically just a place to put money and Register your companies when you have things to hide so, okay. We have developers there
10:45
Just Yes a quick look about the database as I say the database so we have at this time we have a 58 Thousand of ports If you consider we have a three major releases and six architectures that
11:06
brings you a lot of ports, so As I say it's something you can build by your own because I don't have enough space or CPU Also to obtain the list of files
11:21
It's not possible to trust package list because those files are not fully complete sometime you have replacements we don't Do the build and I can just download as if the packages because it needs time and space to
11:43
Download the full list of packages and also have impact on mirror performances. So I found a Simple way is I try to abuse The the package file format as you know packages are just archived who contains
12:04
two two times the manifest of The the package and the second many effects which is a full manifest has the list of files Provide for the package then comes all the files And the thing is you can just use the head
12:25
HTTP request to only download the first part of the package and Let's say we took we can extract the manifest because it works and If the JSON file is valid you're wrong. You're good
12:42
If you if you if you have an error, you just increase a little bit of size Just remember it for the next time. So this way you can just download the only the manifest part of the files and the good thing is the HTTP head is a totally standard method
13:03
So we are not trying to abuse the HTTP server and we are not producing or breakages in connections We are saving people bandwidth Because we are we are not we are not using the the package mirror
13:20
Just for us and we save our also everybody's time Yes, so talking about the database itself at the beginning I Design it as a just as a full text file Just because it's easy to read and search
13:42
But it I have I have this problem. It was too large. It was 36 megabytes Even compress Okay So in 2018 I'd be busy con I talked with talking with Marcus P about the
14:05
Thing he suggests me I can use the locate file format locate is a Unix standard tool who Store all the try to sort all the files you have in your computer. So it's easy to found a place and use a
14:24
Nagorin called big ram to store the the data's and this this file format is pretty efficient because if we once once generated and compress it I reduced by two to not the size of the file, so
14:44
It's it's pretty convenient because Instead of downloading for a couple of minutes the database we can have it just in one one minute and something It's pretty similar with the package itself. So in time
15:00
So the package provide usages as I saw yes, I show you you can just perform reverse searches You're looking for a file give you the name of the package But you can also use it to identify Unexpected the files in the packages like
15:21
core dams things happen sometimes people commit and the commit comes with a binary files or dams because of this happens And also You can use it to search for duplicate files when you have two packages installed the same file and conflicting
15:43
for a long time I report this this information to to the developers and I try to track exactly how much conflicts we have and Someone decide or Stefan Stefan Ester decides to start looking at it and he used the database
16:05
The database I provide to to search for file conflicts and try to fix it To to to to avoid conflict during a package install because package the tool
16:21
Can handle conflicts, but the handle the conflicts the worst way It say okay you have a conflict so or you can't install the new package or you have to remove the old one so there's no way to and sometimes you you have under this a lot of funny things because you have a
16:41
Binary who have to say exactly the same name doing completely different things Sometime is a game and the network manager and Wow so yes, so oh yeah, and I still having six minutes, so
17:01
If you have questions, please Yeah, oh the options no I What I take is the real
17:20
files available in in package Servers so is the real thing you have if you install a package. It's not something. I'm sorry the question was if I Scan for options and the answer is no. I just try to
17:44
track what's on the package Servers and the real files installed if you install a it's not it's not use the custom If you have a custom Package server it doesn't bring you the sort of files you are installing. Yes. It's just for
18:06
Standard instant free BSD packages the way We build it Yeah Oh The first okay, how long is to how long is take to
18:24
Scan the packages and build the index along so the first the first con it's can take about I Think it's I saw I didn't do this for a long time, but I think it's like half an hour
18:41
30 minutes something like that and once Once I build it I can I can just update in a couple of seconds from one minute so it's that the the good thing is if I consider because I We have a file
19:01
Manifesto with All the packages on the version so I can see which package change so I'm concerned I just look at the package who change the version and Based on that and just recovering the Manifesto it takes yes, it takes a cup
19:20
I think the maximum is like a five minutes for for for an architecture and the version Yes The database I used to locate so it's Yes, he's a locate format
19:43
So because because it's it's really it's really convenient for this kind of usage
20:01
Yes, yes, I stored that and yeah Oh Yeah, yes, I I include because the
20:24
The way big RAM works it takes just a line and Just a line is it is a path So it's considered a path and there's no extra data around so you can you can use so I just include
20:41
The the package name inside inside the path and use a special character yes as a separator I'm in this case. I use the the star Character because I don't expect people to put it put a star in the package name
21:01
Yes, absolutely and and I use I use a regular expression for search But the the result is it's quite a bit acceptable In bytes
21:22
Compressed it's 17 exact compress it 70 megabytes. Yes Around because it depends on which version you're looking at, but it's basically yes. It's around this side Yeah
21:42
I Yes could be Could be Yeah, sorry the question. Aha. This was a trick just to see if I can repeat Yes, the question was was why not use a slash instead of star because it's like a slash made more sense
22:02
Probably yes so but I Yes, we can we can probably Oh Good question
22:25
No, I just see happy people
Recommendations
Series of 20 media