Packaging Go in pkgsrc
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 611 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/42253 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Year | 2017 |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2017398 / 611
10
14
15
16
17
21
22
24
25
27
31
36
40
42
46
50
55
56
63
70
73
78
84
94
101
102
104
107
108
109
110
111
112
113
114
115
117
119
122
123
126
127
128
130
131
132
135
136
137
138
141
142
144
145
146
150
151
157
158
159
160
162
163
164
166
170
171
173
175
176
177
179
181
184
187
189
191
193
194
199
200
205
207
208
209
211
214
218
219
222
223
224
225
226
229
230
232
234
236
237
238
239
245
248
249
250
251
253
255
257
258
259
260
261
264
265
266
267
268
271
272
275
277
279
280
282
283
284
287
288
290
292
293
297
302
304
305
306
307
309
310
311
312
313
314
316
317
318
319
321
322
327
329
330
331
333
336
338
339
340
341
346
348
349
350
352
354
356
358
362
363
364
367
371
372
373
375
380
384
385
386
387
388
389
390
391
392
393
394
395
398
400
401
402
405
407
409
411
412
413
416
417
418
420
424
425
427
428
429
431
435
438
439
440
441
443
444
446
448
454
459
460
461
462
465
466
468
471
473
477
478
480
483
487
488
489
491
495
498
499
500
501
502
503
504
507
508
510
511
512
514
518
519
520
522
524
526
528
530
531
533
535
536
549
550
554
555
558
560
563
564
573
575
578
579
582
585
586
588
589
590
591
592
593
594
595
596
600
603
604
605
609
00:00
CodeMultiplication signSource codeComputer programmingData storage device
00:50
Computing platformRootVapor barrierAreaInformationElectronic mailing listVideo gameStandard deviationPatch (Unix)Installation artFormal languageComputer programmingComputer programBuildingGroup actionMaxima and minimaHand fanSurjective functionConvex hullSource codeVariable (mathematics)Computer fileLine (geometry)Computing platformDescriptive statisticsQuicksortCASE <Informatik>Software repositoryOrder (biology)Directory serviceOnline helpMetropolitan area networkMobile appComputer clusterFlow separationBuildingInheritance (object-oriented programming)Entire functionArithmetic progressionElectronic mailing listDot productLevel (video gaming)Installation artPattern languageMeta elementUniform resource locatorBitProjective planeCuboidWindowVapor barrierSubsetComputer programmingBinary codeStructured programmingBinary filePatch (Unix)Data storage deviceInformationConfiguration spaceProgramming languageDistribution (mathematics)SummierbarkeitWeightReading (process)Computer programmingWebsiteSystem programmingTheory of relativityComputer animation
08:05
System programmingMaß <Mathematik>Similarity (geometry)Binary fileSoftware frameworkHome pageSign (mathematics)Simultaneous localization and mappingIcosahedronWechselseitige InformationMIDIConvex hullMetropolitan area networkAcoustic shadowLink (knot theory)Graphical user interfaceCategory of beingHome pageSoftwareQuicksortLink (knot theory)Directory serviceData storage deviceSource codeCASE <Informatik>Entire functionModulare ProgrammierungHexagonComputer fileBinary codeExecution unitBitString (computer science)Exception handlingLibrary (computing)Automatic differentiationBuildingNetwork topologyMereology1 (number)Block (periodic table)Acoustic shadowContent (media)Point (geometry)Software developerIntegrated development environmentInstallation artRevision controlSoftware frameworkNumberEndliche ModelltheorieCodeSystem programming
15:20
Source codeCanonical ensembleMetadataTask (computing)SoftwareAntimatterReading (process)CompilerLibrary (computing)Configuration spacePoint (geometry)Data storage deviceDirection (geometry)CircleRevision controlSource codeDescriptive statisticsQuicksortGoogolPoint cloudMetadataCanonical ensembleComputer fileBuildingWeightWeb 2.0Wrapper (data mining)Server (computing)Reading (process)Projective planeEncryptionAcoustic shadowUsabilityNetwork topologySoftware testingFunction (mathematics)Level (video gaming)CodeLinear regressionCountingSoftwareAuthorizationProcess (computing)MereologyDirectory serviceUniform resource locatorImage resolutionRow (database)Binary file1 (number)Link (knot theory)Software repositoryLine (geometry)Installation artGoodness of fitXINGFitness functionAnalytic continuationComputer animation
22:35
GoogolPoint cloudQuicksortLink (knot theory)Medical imagingSoftware bugData storage deviceScripting languageProcess (computing)Directory serviceComputing platformWeb pageEvent horizonDescriptive statisticsWeightFeedbackSource codeReverse engineeringCloud computingSlide ruleBitMultilaterationComputer animation
24:43
Computer animation
Transcript: English(auto-generated)
00:00
Okay, hello everybody. I would like to welcome you to the BSD Dev Room at FOSM 2017. Thank you for being such a large crowd here. I see a lot of faces I know, which is good. I'm gonna tell you as our first talk today in this
00:23
Dev Room about how we package source code of programs written in Go in package source. So first of all, I would like a quick show of hands, who here knows what package source is? Okay, so I don't need to explain it for a long time.
00:41
Who here knows about Go? Most people? That's great. It's awesome. So package source, obviously, is the NetBSD packages collection, which doesn't mean it runs only on NetBSD. It means it's maintained by people in the NetBSD project.
01:01
It contains over 17,000 packages. It's been growing. It's still growing after all these years. And the really cool thing about it is that it runs on 23 different platforms out of the box. So that includes any sort of BSD you might want to throw at it, like NetBSD, OpenBSD, FreeBSD, Dragonfly, Bitrig, whatever.
01:27
It includes Linux, even Windows. The Bash for Windows thing that they're doing is basically Linux and Cygwin, I believe, is supported as well. Some cool features that package source will give you.
01:42
You can, if you're on one of the mainstream platforms, you can use pre-compiled binary packages. There's something called package in, which is a lot like apt-get. So really easy installation, upgrade, and so forth. You can set it up unprivileged. That can be super, super helpful.
02:01
That means you don't need root for building packages or installing packages or using them. So you could do a container-like thing if you wanted to, like poor man's Docker, so to speak, by having a private package source tree per major app that you just package together and it doesn't need root access
02:22
to do anything, which is cool. And there is a separate side project, it's called WIP, work in progress. So if you want to get started with package source, after all that I've told you today, it's super, super easy to get an account for that. It uses Git, unlike main package source,
02:44
which uses CBS still. And it's very low barrier to entry, and it's really, really very easy to create packages. To go a little more into detail, a package in package source minimally needs four files. There are makefile, distinfo, descur, description,
03:04
and the plist, the packing list. So the makefile is essentially a bunch of variables that are being declared and then at the end you include one include line that does all the work. The distinfo is automatically generated.
03:21
It contains checksums for the file you're gonna download. Description is something you need to write. It's, well, it's one or two paragraphs describing the package, and the packing list, again, is auto-generated by the infrastructure is a list of files in the package, basically. And the installation of a package source package
03:44
or the build goes in these predefined stages. So that's a structure that's basically set in stone. And it's, but it's worked really well over the years, and it's a good structure. So I probably forgot a bunch of stages here, but first of all, the first stage is fetch,
04:03
where you download the distribution file. Or you know, you copy it from somewhere where you have it in your local repo. Then you check the checksums to verify that you've actually got the correct file with the correct size.
04:21
Then you build tools, you install dependencies that goes recursively, does the same thing for every package you depend on. Then you extract the source code. Then you apply any patches on top of it, if there are any. Then there's a stage called build where you do the actual, I forgot configure, obviously. There's configure, then build, then stage install,
04:42
which is where you install it into a staging directory, the so-called destir, and then the package stage, which takes whatever is in the destir, or rather, the subset that's in the plist, and packs it together in a package.
05:01
And then package install takes that generated binary package and installs it on your system, right? Now we switch gears a little bit and go into the world of Go. So it's a fairly popular programming language, to put it mildly.
05:21
It's, in fact, it's so popular that tomorrow there's gonna be an entire dev room about Go-related topics in the DePaz room, in the H building. In the world of Go, there is a tool that's also called Go, and the Go tool handles, among other things,
05:42
fetching, building, and installing Go programs, and let's look at how it does that. Here's a simple example. So I chose a random package that was the last package I was working on. Don't read too much into it. So what's happening here is you tell it to go get,
06:02
something, I'm sorry, it's a little bit small. The three dots at the end is a meta pattern. It's like a star, except it matches subdirectory. So that means I want to install everything
06:22
that's under this path, download it, and install it. And so go get will download that. You notice import paths look like URLs. That's intentional. Then it sees that there is a dependency in there that we don't have, so it tries to download that.
06:40
There's this little bit of a special case because it's sort of a sim link, so to speak, from an import path to an actual Git repo somewhere else. Then it downloads that, and then it builds the things in opposite order. Sounds similar enough to what we did in package source.
07:04
And there's this thing called the go path at the top. If we look at this go path that we've just created, you'll notice that there are no two directories called source and package. There might be a third directory called bin if there was a binary. And so source contains a source code
07:21
for those things that we downloaded, and it's arranged by import path. That's a standard way that go arranges its source code. And then the package subdirectory has another subdirectory for your platform. So from the same source tree, you could build for as many platforms as you want,
07:42
and it would not conflict. And then it has, you'll notice it has one level of directories less because each subdirectory in source is conceptually one package corresponding to one file in PKG. Now there is some mismatch between that because it does download, then discover something,
08:03
then download some more stuff, then discover some more, and then build stuff recursively in the middle. So that doesn't quite fit with the model we have in package source. Now there's two possible approaches to this.
08:21
One of them is to basically ignore the ecosystem, to say we only package go itself, and then tell the user, just run go get something something, and you'll get your binary. And in fact, that is what the people from the go team have recommended to me. But I think that's unsatisfactory.
08:40
That's clearly not what you want because at some point there is some software that's written in go that somebody wants to run, and you don't want them to be special snowflakes. You want them to be packages such as all the rest of your system. In package source, for example, we have a linter called package lint that's written in go.
09:01
So it's actually become quite an important topic for a bunch of developers. So let's deal with it. Some more thoughts on this is most of the packages or the individual, say, downloadable units in the go ecosystem are kind of small and do relatively little.
09:22
And in that they're similar to Perl packages or maybe the tech live ones or so. So there's like hundreds of tiny ones. Or if you've ever used Node.js and you've used NPM, like you install some random tool, you do NPM install bower, for example, and then you look in your NPM modules directory
09:42
and all of a sudden you have 350 modules in there. And it's a little bit like that. Now the binaries, once they're built, they don't depend on anything really because they're statically linked. Except maybe libc, but okay. And so once you've built your binary package, you can actually install it
10:01
without having all the intermediate libraries there. So that's kind of a nice property. So we have a framework in package source called go package.mk that allows you to write these packages relatively easily. We're going to go through this go-ovh that we've seen before. See how that works. So first step when you wanna package this thing,
10:20
we go to its homepage or to its GitHub in this case, and we download the latest release. Oh look, there are no releases. Yeah, okay, so instead we search for the latest commit and take that date and take the SHA-1 of the commit,
10:40
which is an endless hex string. And then we stick that in our makefile. Here's the actual makefile for this thing, except there's one bit missing. So you notice the version number's a little bit awkward. We've chosen a category. We have this weird tag thingy here.
11:02
And then the more interesting bits for Go specifically is probably this block here. So the go source path is basically where in your go path this thing would be placed.
11:21
So it's the import path you want for that package in the end. And the go dist base is the directory name that your GitHub download will have, because if you just extract the file, it will not be in the correct directory. So this does a little bit of gymnastics to move it into a newly created go path, so to speak,
11:43
to this subdirectory. And then you build, and it looks like this. Again, apologies for the font size. And there's, this here happens. Well, of course, there was a dependency. We don't have that package. But wait, you say, I have that package from earlier,
12:01
right, I downloaded it. But yeah, you don't want that. You want your build to be hermetic in the sense that it uses only dependencies it has actually declared so that in the end your dependency tree makes sense. So it's clear that it's a good thing
12:22
that it's not finding it. Now, let's add the dependency. We add an include for the build link 3.mk of this package which conveniently enough already exists. Then we run make and you see kind of unspectacular, not very spectacular, it builds.
12:42
So what happened now? What did this include build link something, something actually do? So build link is a framework part of package source. It was written originally for, I suppose, the package is written in C or C++.
13:02
Many of these use autoconf. And if you're trying to package something that uses autoconf, it checks for presence or absence of certain libraries. So if you happen to have an extra library installed that the package may link against,
13:20
it can add that as a dependency and then it's not recorded in your package but the binary does depend on it. So that's a big problem. I think OpenBSD had a long history of sort of purging unwanted dependencies. Package source did this build link thing instead and it turns out it works beautifully for Go
13:42
because what we do, we create a shadow Go path where we link in the entire source code and also the package files, like the entire contents of all the declared dependencies.
14:03
And then we set the environment variable to tell the Go tool that it should look there for code. And that works just fine. I'll show you how the build link 3.mk works. In this case, I'll show you the one for this package because we wanna allow other packages
14:21
to depend on it, obviously. There's a little tool called create build link that basically creates this thing on its own. So what it does, this is the magic bit here, the build link content filter. So it takes the entire contents of the go.pkg subdirectory which is where we install the source
14:40
and pkg directories and build links it in. The dependency method is build so that means it will not be a dependency of the final package. It's only a build dependency, which is what it is. And here it has, we have repeated this include.
15:01
So if you depend on go.ovh, it'll transitively depend on go.ini so we can actually build the thing. This has another very nice property and also a very important property, the compiled source code. So the way the Go tool works is a little bit like make.
15:28
If any of the source files are newer than the binary file, then the binary file is stale. So it needs to be rebuilt. If your compiler is a different version than the compiled file, the compiled file is also stale, needs to be rebuilt.
15:42
However, you can't just rebuild random things that are already installed because their file checksums have been recorded. And also the directory is hopefully read only. So when you build link the tree in, then it can replace those compiled files
16:02
with newer versions if it needs to. And it doesn't count because you just delete the shadow go path once you're done. So that is actually, that solves a lot of problems and it makes it kind of lower the mismatch between package source and go.
16:24
Now, I would say, so this is basically where we are, but the problem is it's still relatively tedious. So to give you an example, because those anecdotes are totally what we need. To give you an example, I'm working on something called Caddy. It's a web server written in Go. Very high performance, very cool project.
16:43
Encryption built in, zero config and so on. Has hundreds of other Go packages that it depends on. And if you package them, one by one, then you're gonna be like me two months later still packaging things. So yeah, this could be automated, right?
17:02
As the next logical step in this direction. However, it's not trivial. You've seen a lot of metadata that I've manually added basically. Like, you know, because there was no release, the finding the latest commit and date and so on and putting that in. The comment which I've mentioned
17:20
is a one line description of the package. I don't know where to get that from. What's the license of the package? There's no, so unlike Perl packages, there's no manifest that has these things recorded in a neatly consumable way, which is very sad.
17:40
Then the long description. Readme.md files sometimes leave a lot to be desired. You can't just take the first few paragraphs and say this is my description, it doesn't work well. Then the whole canonical import path versus source code location that the tooling can help you. So you saw the Go tool discover where the thing actually is.
18:01
You can use that programmatically. The same way you could do dependency resolution programmatically, you can ask the Go tool which other packages does this thing depend on, but please filter out the standard ones. And then you match that against packages you have already in your tree so you can write the depends files.
18:21
That would be maybe a worthwhile first step. Then some of those have extra dependencies for testing. Unfortunately, package source doesn't have a concept of what OpenBC called regress depends. So you can't say package x, y is only used for the regression tests.
18:41
The way I've been doing that is I put it in the build dependencies in the make file but not in the build link file. But it's probably not the greatest solution. Sometimes things depend on C libraries like you have Go wrappers that wrap a C library.
19:00
Yeah, so I've previously written such a thing for Perl packages actually. It was called C pen to port or C pen to package on the package source side. So maybe I should tackle that at some point. Would be a worthwhile thing to do I think because the Go ecosystem is large and has many interesting things that may not be,
19:21
I would say may not be known enough in the circle of people who were likely to use say NetBSD. But there's some cool stuff in there. And so as my last sort of point I'm trying to make and that I also promised I would make
19:41
in the description of this talk is as an upstream author, like as the author of a Go package, what can you do to make it easy to package the software that you have written? So my first point is give us releases please. A lot of the Go community never releases anything.
20:01
They just push on GitHub continuously. And if you do releases, please also do release notes because we like seeing what changed. Then we've sort of tackled the whole Go get process. So it would be nice if your software supported that.
20:23
And so I've seen things, I don't remember what project this was, it delivered an install.go and said just compile and run this thing and it'll do the rest. This is not likely to work, unfortunately.
20:40
And the most horrible part is when your software during builds, for example this install.go, during build decides to download more stuff. That does not fit at all with the hermetic building that we would like to have. And that is just necessary for making
21:04
any sort of reproducible package out of your code. So don't do this please. And also please, some read me's are really bad as I said. Please put more sensible descriptions. What does your thing actually do? Maybe for just libraries it might be more fruitful
21:22
to look at the Go doc output because many people have crappy read me's but good comments in the code that might be an idea. And avoid circular dependencies. That's a good one as well. So the go get works on the level of package, like it builds package by package.
21:42
Whereas package source typically works on a per dist file. So basically all of, for example, go lang.org slash x slash net is one entity for us. And there used to be a circular dependency, might still be in there. There's a package in go net that depends on
22:03
the metadata package from the Google Cloud libraries. And then there's another package in the Google Cloud libraries that depends on something from the x net repo. So if you try to build first one then the other,
22:22
you're gonna fail. So you have to split one of the two to get rid of the circular dependency. And that's really crappy so please don't do it. And that's mostly it. Thank you, here are some links. The first one is the link to this event. You can leave your feedback here.
22:40
I'll upload the slides a little bit later. Then package source org, don't need to explain, go lang org, also package source SE is super cool. It's a browsable package directory of the things that are in there. So you can search for something like go ovh or whatever and it will pop up a little page
23:01
that gives you the description, gives you a link to the sources. You can click on the dependencies, reverse dependencies, that's really nice. And I have one bonus slide. Maybe if there are some questions first. Are there any questions? No. Okay, here's the bonus slide.
23:21
So NetBSD is kinda cool if you're using cloud stuff because of its Xen support. It runs great on a bunch of Xen-based cloud solutions such as Amazon EC2. And starting from NetBSD 7.1, our release candidate one,
23:43
we finally have enough stuff to be able to run on the Google compute engine which is sort of KVM-based. However, there are still a couple of bugs. It's not quite easy to create such images. So there's semi-official support coming.
24:03
This is something I've been doing in my day job. It's not released yet. That URL doesn't work yet. It'll be there in one or two weeks. And it's basically a script where you can enter a release and the platform 386 or MD64, it builds you an image and then it can upload
24:22
and there's a readme that explains how to use it. You can upload that to Google Cloud Platform and create your own NetBSD VMs. Thank you very much.