Free Software, Dependency Management, and what I got wrong at FOSDEM 21
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 287 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/57088 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2022247 / 287
2
4
6
8
12
17
21
23
31
35
37
41
44
45
46
47
50
62
65
66
67
68
71
73
81
84
85
86
90
92
94
100
102
105
111
114
115
116
117
118
121
122
124
127
131
133
135
137
139
140
141
142
145
149
150
156
164
165
167
169
170
171
172
174
176
178
180
183
184
189
190
192
194
198
205
206
207
208
210
218
220
224
225
229
230
232
235
236
238
239
240
242
243
244
245
246
249
250
253
260
262
264
267
273
274
277
282
283
287
00:00
Programming languageMultiplication signFormal languageData managementFilm editingFeedbackSelf-organizationQuicksortFreewareSoftwareDirection (geometry)Ideal (ethics)BitComputer animation
02:33
Computer programmingGoogolOpen sourceFreewareCodeIntegrated development environmentPhysical lawSoftwareMusical ensembleQuicksortFormal languageSoftware developerProcess (computing)Projective planeSoftware maintenanceComputer animation
04:48
FreewareSoftwareAreaQuicksortSpectrum (functional analysis)Endliche ModelltheorieAxiom of choiceSoftware developerComputer animation
07:28
Software engineeringRevision controlGoogolProgrammer (hardware)Computer programmingOpen sourceFreewareCodeMachine codeSoftware developerSoftwareRun-time systemElectronic mailing listInternetworkingComputer animation
08:41
SoftwareComplete metric spaceTerm (mathematics)Coordinate systemElectronic mailing listPerspective (visual)Chemical equationQuicksortInternetworkingMultiplication signDifferent (Kate Ryan album)TheoryProjective planeFreewareCodeData conversionWhiteboardAuthorizationLibrary (computing)Spectrum (functional analysis)MathematicsScaling (geometry)Web pageSelf-organizationInfinitySoftware developerComputer animation
14:01
Gamma functionStreaming mediaUtility softwareSoftwareFunctional (mathematics)AuthorizationSpectrum (functional analysis)TypprüfungSlide ruleComputer fileType theoryContinuum hypothesisQuicksortFluid staticsProcess (computing)CASE <Informatik>Revision controlChemical equationProduct (business)Electronic signatureCodeData managementBasis <Mathematik>Statement (computer science)Similarity (geometry)Normal (geometry)Different (Kate Ryan album)Computer animation
19:21
Johann Peter HebelPoint (geometry)Computer animation
19:39
Product (business)Projective planePerspective (visual)Multiplication signView (database)QuicksortPhysical systemAuthorizationBitData conversionEntire functionExpected valueComputer fileWordProgramming languageSet (mathematics)Computer animation
22:07
Computer animation
Transcript: English(auto-generated)
00:12
Hi, my name is Daniel Sockwell, although some of you may know me better by my username CodeSections, which is the username I use on the AccuIRC channel or on the subreddit.
00:25
Feel free to reach out to me in either of those places or anywhere else, or at my website codesections.com if you'd like to discuss this talk afterwards, I'd love to hear your thoughts. And today's topic, I'm going to be talking about dependency management and sort of more broadly,
00:47
lessons that Raku can learn from other programming languages that are more targeted at solving the problems of large software organizations.
01:04
In particular, because of a talk I gave at FOSTM last year, and I'm hoping to save a fair amount of time from this talk to take questions and hear your thoughts. This is a topic that I've been giving a lot of thought to over the last year,
01:24
but it's also one that I still feel like I don't have figured out, so I'm very much hoping to hear other people's thoughts and feedback as well. So, diving in, last year I gave a talk entitled
01:44
Imagining the Ideal Programming Language for Writing Free Software, and I talked about how Raku is a really great language for writing free software, and I contrasted Raku, which I said is ideal for small groups of programmers,
02:06
perhaps even individuals, writing free software to big teams working for a large corporation. And I stand by what I said there, but I've also had some thoughts that sort of cut in the opposite direction.
02:23
So, to say exactly what I mean, I want to play a clip from my talk last year, and then talk a little bit about what I still believe and what I don't quite believe. So, here's that clip. But, I think that that environment is very different from the sort of solo developer,
02:43
free software, or code as novel environment that is really important for a free software language to target. To explain what I mean, I'd like to look at three different features of the big tech environment
03:02
compared to the free software, code as novel environment, and then for each of those features, look at how that feature of the environment impacts the language design goals of a language targeting that sort of environment. So, first, in big tech, you have really large teams, small teams in free software,
03:27
as we've talked about pretty extensively. Big tech also has high turnover. Just like the law firm I worked at, it's not at all uncommon for people to start their career there before quickly moving on to somewhere else,
03:41
and they're sort of constantly hiring from people who have just completed their CS education. Free software, on the other hand, a lot of these small projects have one maintainer for decades, but even if it's not quite to that extreme, the turnover is much, much lower.
04:05
In big tech, people are there because they're getting a paycheck. They may have other motivations as well, but that is a sort of defining feature of the job is that it is a job. Free software developers tend to be much more self-motivated.
04:22
They may or may not be getting paid at all. If they are, they're getting paid probably less than they would make at a different way that they could be spending their time, so the motivation is pretty different. So how do those features of the big tech environment impact design goals
04:43
of a language targeting that environment? So basically, my model last year was that there was a spectrum
05:02
that looked something like this, where you have a big company, a big tech company, that in every way is on sort of the opposite end of the spectrum from a free software project, whether that's team size or turnover or motivation.
05:23
Big tech, huge team, high turnover, monetary motivation, it's on the exact opposite end of the spectrum of a free software project, and then some sort of small company is somewhere in between those two extremes,
05:42
but very much sort of tilted towards the big company side, where you still have a significant size team, you still have a significant amount of turnover, you still have a significantly monetary incentive base for the work you're doing there.
06:04
And so those are both sort of on the same side of the spectrum generally, and free software is sort of the odd one out, and thus some of the design choices that Raku makes are motivated by that way
06:25
in which free software is on sort of the opposite end of the spectrum. And I still generally believe that, and I stand by pretty much everything I said in that talk, but I've come to believe also that there are at least a few ways where that's less the case,
06:47
where instead of free software being on the opposite end of the spectrum, free software is on the same end of the spectrum as big tech and the opposite end as a small company, but free software is not opposite of big tech,
07:04
it's big tech, but even more so. And to show an example of one area where I think this is true, and the area that got me thinking about this is dependency management,
07:20
so let me play a clip from the talk that sort of clued me into this, which is a talk by Russ Cox, one of the leading developers of Go, talking about Go as impaired to free software, and how Google's software environment relates to a free software environment.
07:43
Versions are absolutely necessary for software engineering. They are how you explain both to tools and to other programmers which code to use, even though that code is changing over time. Versions are especially necessary in large-scale software engineering, where there are many other programmers, and the code has to keep working for a very long time.
08:03
Now, when I say large-scale software engineering, some people think of programs at a large company like Google, but in many ways Google is small. We have all our code in one place. We can find all uses of a given package. We can list all the programmers working on a given program. In contrast, consider open source development on the Internet.
08:24
The code is not all in one place. We can't find all the uses of a given package. We can't list all the programmers working on a given program. This is the setting where we most need good support for versioning to simplify software engineering.
08:42
So, to expand on what Russ Cox was saying there a little bit, you know, imagine that in a small company, you might have a team of five or six other developers who use the library that you're writing. If you want to make a breaking change to that library,
09:02
you can sit down in a room with those five or six people using your code and tell them, you know, this is the change I want to make. Can we all get on board and all upgrade at the same time and everyone stop using the old version? And you can sit around and have that conversation.
09:22
In a big tech company, it's theoretically possible to do that, to have that same sort of conversation, but much more challenging, you know, instead of five or six people there. If you listed out all of the different departments and different teams that use the library that you're working on, it may be that there are 20 or 30 different teams
09:43
and each of those is not just one person, but is, you know, a different team within the organization. If you tried to get everybody on board, it would be a huge logistical challenge in practice. It's going to be very difficult to coordinate upgrade paths,
10:02
but in theory you could do it. Free software, though, takes that practical difficulty of big tech and makes it actually a theoretical impossibility, you know. Even if you were willing to go to a sort of infinite amount of trouble, it's just even in theory impossible to figure out,
10:25
to have a complete list of everyone who is using your project. You know, someone can have downloaded it from the internet and they are under no obligation to tell you. So, like, if you, there is no, even in theory,
10:41
there is in principle no way to have a complete list of all of the people for whom your software is a dependency. So, you know, from that perspective, when looking at how you manage your dependencies, or rather how you think about your software
11:04
from the perspective of who might be depending on it, unlike all of the ways I talked about last year, free software is not on the opposite end of the spectrum from big tech. Free software is facing the same problems as big tech,
11:24
but turned up to 11, or from some perspective turned up to infinity. You have a problem that is very hard to solve and big tech is even in principle impossible to solve
11:40
for free software in terms of listing everything, in terms of managing all of the users of your software. Why does that matter for dependency management? Well, I think it matters because of some of the problems
12:03
that Russ Cox goes on to talk about in that talk about Golang and specifically specifying dependencies in a way
12:20
that allows a sort of clean upgrade path when you, as a user of a dependency, don't have a way of getting on the same page with all of the other users of a particular dependency, or from the other perspective, when I, as the author of a library, don't have a way of coordinating with all of the users of my library.
12:47
How does this relate to Vaku? Well, I think it relates in terms of the balance between two aspects that are very important to Raku,
13:10
and specifically I'm thinking about the balance between what is sometimes called whip-uptitude, the ability to just easily whip up a script and get started with Raku right away,
13:22
and what you could call software engineering, the ability to make sort of very solid software that works regardless of, or not regardless, that's the wrong way to put it, that is solid and dependable
13:42
and will work at a much larger scale. So one way that Raku most obviously strikes that balance is you can just start declaring your code and just jump in right away,
14:01
or you can declare types and have some of the benefits of more static type-checking type code by declaring your types and declaring your function signatures
14:22
and function return values and all of that. So in general, Raku does a very good job of striking that balance where you can start by just whipping something up, and then as it progresses and becomes bigger, you can sort of slide along this continuum
14:42
to as much rigor and as much sort of locking things in as you would like. And so I think that we do something similar with dependency management in a way that is not talked about
15:03
quite as much in our documentation and a way that I would like to draw more attention to, but also in a way that is conceptually at least very similar to what we do with types
15:21
and signatures and all of that. So for example, when importing statements, you can just use a package. So I recently released the low-bar utility package, and the easiest way to use that
15:42
is to just put the name of the package, in this case just an underscore, and just use it. But much like Golang's full import paths, we can also specify an exact version,
16:01
if I can type, an exact version. We can also add an auth to it. So we have the ability to specify on a per-file basis
16:25
the exact version, the exact import path of each dependency, and that gives us the ability to be very specific,
16:41
but also a bit verbose. You know, there is a big difference between just having a simple use statement versus a use statement that is
17:01
as ver something, and then maybe auth something, and we can add additional qualifications as well.
17:22
So I guess it's up to the individual, the authors of each individual file to decide where on this spectrum they want to place their dependency management.
17:43
But what I'm struggling with and what I would like to hear people's thoughts on is what our community practices ought to be for this spectrum.
18:00
Right now, the vast majority of the examples in our documentation have simple use statements that are unqualified. That's very much on the whip-uptitude side of the spectrum. My instinct after thinking about this is that that is good for simple scripts,
18:22
but that we really ought to be encouraging people at the very least to specify full versions and perhaps to also specify full import paths on anything that they intend to be more in the production side.
18:40
And I'm also sort of giving thought to what practices we ought to have for encouraging, sort of versioning best practices and that sort of practice for software authors.
19:03
Obviously, that's the sort of thing that happens through sort of ecosystem community norms and not through any sort of tooling or any sort of official mandate,
19:20
but I think that that is equally important, and that's one of the points that comes up in this talk is cooperation is something that is absolutely key to having a well-functioning ecosystem. It's not something that you can sort of
19:40
legislate or enforce through the programming language itself, but it's something that emerges out of the ecosystem and out of the practices as practiced by individual authors and by individual users
20:02
and what the sort of community set of expectations is and what sort of accepted best practices are. So that's something that I'd be interested to hear people's thoughts on. And really, I would like all of this to be as much a,
20:26
as I said, as much the springboard for a conversation as it is for, as it is me sort of talking at you, which is why I really wanted to leave a lot of time for question and discussion.
20:44
So if there's any takeaway from this, I'd like you to take away the idea that we don't have just one versioning system in MACU. We have an entire spectrum, but that for production software,
21:00
I would encourage people to be a bit more on the explicit side of specifying their dependencies, not just on a per project basis, but actually on a per file basis, but that it's something that we really ought to
21:22
not just take my word for, but come up with, as a community, come up with some thoughts about what our sort of consensus best practice is and maybe even document that, or at least articulate it through our documentation in a way that does sort of express
21:41
what that consensus view is. And I'd love to hear your thoughts on that. So with that, I'm going to stop talking and hope that some of y'all will chime in with some thoughts and some perspectives on this,
22:00
and maybe we'll reach a bit more of agreement as a community.