The Trouble with FreeBSD
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 31 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/45251 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
1
3
5
6
7
9
10
11
13
14
15
16
18
19
21
22
26
27
28
29
30
00:00
Video gameWordIn-System-ProgrammierungPowerPCProcess (computing)Virtual machineBitPatch (Unix)Continuous integrationSoftware testingUniverse (mathematics)BuildingOpen sourcePresentation of a groupVideo gameAutomationPoint (geometry)Suite (music)QuicksortFormal languageForm (programming)Computer animation
03:05
Repository (publishing)Line (geometry)Multiplication signRepository (publishing)Open sourceLine (geometry)MereologyVirtual machineCycle (graph theory)Order (biology)Computer fileDisk read-and-write headCodeProjective planePoint (geometry)Entire functionSoftware testingNumberMiniDiscStandard deviationSource codeComputer animation
04:50
Unit testingProjective planeKernel (computing)CodeMultiplication signWritingExecution unitSoftware development kitScaling (geometry)Software testingNumbering schemePatch (Unix)Source codeElement (mathematics)MathematicsArithmetic meanTouch typingComputer animation
06:05
Source codeRemote administrationEmailSoftware bugDistribution (mathematics)Tape driveLoginProjective planePoint (geometry)WordGUI widgetVirtual machineRepository (publishing)MathematicsCore dumpElectronic mailing listGroup actionPatch (Unix)CodeOpen sourceComputer animation
06:57
ArchitectureDecision theoryCore dumpRankingGame controllerTerm (mathematics)Goodness of fitComputer architectureDirection (geometry)CASE <Informatik>CausalityComputer animation
07:54
Core dumpOpen sourceCartesian coordinate systemScripting languageGastropod shellTrailSource codeAxiom of choiceRevision controlCore dumpMultiplication signGastropod shellMathematicsScripting languageBitSinc functionWeb crawlerPhysical systemProcess (computing)Computer configurationElectronic mailing listRepository (publishing)Endliche ModelltheorieAuthorizationProjective planeTelecommunicationEmailSoftware bugLevel (video gaming)CASE <Informatik>QuicksortOpen sourceMultilaterationComputer animation
10:56
QuicksortMechanism designLengthProjective planeComputer fileBranch (computer science)MathematicsStability theoryStructural loadEndliche ModelltheorieRevision controlInteractive televisionControl systemPressureWrapper (data mining)CD-ROMMultiplication signRepository (publishing)Point (geometry)Data storage deviceFreewareTransformation (genetics)Process (computing)Forcing (mathematics)Task (computing)Streaming mediaComputer animation
13:36
Thread (computing)Message passingOpen sourceUniform resource locatorComputer fileSoftware repositorySemiconductor memoryEndliche ModelltheorieEntire functionProof theoryRepository (publishing)Standard deviationKey (cryptography)Matching (graph theory)State of matterData conversionRevision controlProcess (computing)Software developerProjective planeCodePhysical systemMultiplication signRight angle1 (number)Computer animation
15:44
Multiplication signProjective planeMathematicsSource codeCore dumpSoftware developerProcess (computing)Network topologyComputer animation
16:39
Pauli exclusion principlePauli exclusion principleGroup action1 (number)MathematicsProcess (computing)Information technology consultingResultantEntire functionFocus (optics)Point (geometry)Theory of relativityPressureData structureEmailCore dumpLevel (video gaming)Similarity (geometry)FeedbackDecision theoryJava appletDifferent (Kate Ryan album)Text editorNumberProjective planeCodeSound effectVideo gameElectronic mailing listStandard deviationExtension (kinesiology)Computer animation
19:19
Electronic program guideComputer-generated imageryMachine visionDifferent (Kate Ryan album)Open sourceOnline helpPeer-to-peerCodeVideo gameSoftwareSoftware maintenanceInformation securitySoftware testingImage resolutionElectric currentDirectory serviceLevel (video gaming)Software developerCryptographyElectronic mailing listSoftware developerMathematicsDifferential (mechanical device)1 (number)Theory of relativityCore dumpSheaf (mathematics)CASE <Informatik>Electronic mailing listMedical imagingRule of inferenceSubsetProjective planeFreewareCodeElectronic program guidePosition operatorSystem administratorSoftware repositoryMultilaterationThermal conductivityData managementWeb pageFlow separationStability theoryInteractive televisionBitNP-hardNumberOnline helpQuicksortAreaOcean currentMetropolitan area networkCommitment schemeSource codeJSON
22:23
Latent heatMathematical optimizationPower (physics)Sheaf (mathematics)Computer architectureMultiplication signBitDifferent (Kate Ryan album)QuicksortKernel (computing)Process (computing)FluxProjective planeCodeFilm editingMultiplicationInterrupt <Informatik>SpacetimeTrailAreaCASE <Informatik>Parameter (computer programming)MehrplatzsystemCross-platformMathematicsDisk read-and-write headGame theoryLevel (video gaming)Software maintenanceSoftware developerRevision controlVirtual machineFeedbackHeegaard splittingoutputGoodness of fitEmailCore dumpFreewareMehrprozessorsystemHill differential equationOnline helpComputer animation
27:50
Core dumpCodeInformation securityImplementationFormal grammarVotingSoftware developerNormal (geometry)Revision controlRule of inferenceTerm (mathematics)Decision theoryType theoryPointer (computer programming)Computer fileMathematicsStrutOvalQuicksortProjective planeCore dumpFlow separationRule of inferenceSystem callElectronic program guideMultiplication signEmailCodeFile archiverPoint (geometry)WindowData managementCommitment schemeSoftware developerBranch (computer science)Interactive televisionFreezingBitGroup actionStability theoryElectronic mailing listMathematicsDynamical systemLinker (computing)Image resolutionThermal conductivityOnline helpGame controllerCASE <Informatik>Ocean currentPattern languageSoftware maintenanceMereologyInformation securityVapor barrierKernel (computing)Process (computing)Message passingInternet forumOffice suiteDependent and independent variablesRight angleTelecommunicationForm (programming)Revision controlDifferent (Kate Ryan album)FreeBSD 4Computer animation
33:43
Core dumpProcess (computing)Game theoryStatement (computer science)2 (number)Software developerInclusion mapEmailLoginThermal conductivityScripting languageElectronic mailing listTwitterProjective planeCodeSystem administratorMathematicsFreewareInstallation artCore dumpTheory of relativityGame controllerPoint (geometry)QuicksortConnected spaceFigurate numberFormal grammarRule of inferenceMultiplication signGraph (mathematics)HypermediaSoftwareExtension (kinesiology)BitOnline helpType theoryRevision controlExploit (computer security)Instance (computer science)Correspondence (mathematics)Arithmetic progressionComputer animation
39:00
CodeProjective planeMereologyCodeBuildingProduct (business)Core dumpThermal conductivityBitSoftware developerRevision controlWordMessage passingSpacetimeGroup actionDifferent (Kate Ryan album)BlogFacebookTwitterInformation technology consultingComputer animation
41:43
BitRight anglePhysical systemMathematicsCycle (graph theory)Euler anglesEmailBuildingProjective planeParameter (computer programming)CodeMultiplication signHeegaard splittingQuicksortSoftware testingTerm (mathematics)CASE <Informatik>SpacetimeComputer animation
43:17
SpacetimeHuman migrationQuicksortMultiplication signCore dumpProjective planeDecision theoryRule of inferenceReal numberProcess (computing)Right angleMathematicsSet (mathematics)Revision controlDifferent (Kate Ryan album)JSONComputer animation
46:10
Information technology consultingCodeGroup actionCycle (graph theory)Term (mathematics)Open sourceEmailRational numberParameter (computer programming)Projective planeMathematicsMultiplication signGoodness of fitDecision theoryMereologyReal numberThermal conductivityProgrammer (hardware)Statement (computer science)QuicksortUltraviolet photoelectron spectroscopyThread (computing)Level (video gaming)Process (computing)Core dumpPoint (geometry)Library (computing)Operator (mathematics)Moment (mathematics)BuildingSpacetimeIntegrated development environmentWave packetTwitterWordSimilarity (geometry)Revision controlBoss CorporationSet (mathematics)Computer programmingVirtualizationEntrainment (chronobiology)Different (Kate Ryan album)Reduction of orderComputer architectureDirection (geometry)Hidden Markov modelComputer animation
Transcript: English(auto-generated)
00:05
Okay, thank you for coming to my lovely talk. Thank you, thank you. So in Australia, there's a yearly conference called LinuxConf AU, and despite the name, it's more of a generalist open source conference. Occasionally, I have discussions about changing the name
00:21
and no one can ever agree on a new one, so they just keep it. In 2016, it was in Geelong, which is a town just outside of Melbourne, and there was a presentation there by a woman named Emily Dunham. And she was presenting, she works for Mozilla on the Rust language, and she gave this excellent
00:41
presentation on how Rust used various forms of automation to make their community life better. So one of the things that she pointed out was, if you have an automated test suite, automated sort of build continuous integration set up going, it's a lot easier to take rejection from a machine than it is to take it from a human.
01:00
If a human tells you your patch is bad, then you feel offended. If a machine tells you your patch is bad, you go, why did you say that machine? And you go and make it better. And it was all kinds of little things like this that they used to try and make Rust a really welcoming place and a really happy place to be. And I sat through this whole thing going, A, that's awesome, and B, why can't we have these nice things?
01:21
And so, who was that? And so I sat down and went through and tried to work out why exactly FreeBSD can't have these nice things. And so here we go. So just a bit of background for anyone who doesn't know me. I got involved in FreeBSD first back in the mid 90s working at an ISP.
01:41
I discovered Linux and some friends at my university choir of all places told me, don't use that Linux thing, use FreeBSD. So I did. And then later on, I decided I wanted to work out more about how it worked. And so I decided to jump in the deepest end I could find and got myself an iMac and tried to get it running on that. This was back when iMacs used PowerPC.
02:02
And then nearly 18 years after I started using it, I actually got a job where I was actually getting paid to do it. So just as a bit of a warning, this is going to be a very talky talk. There's not a lot of funny pictures in this. There are occasionally some bits of text, but that's mostly what's gonna be up here is it's gonna be lots and lots of text.
02:23
I'm also going to be airing out some bits of FreeBSD's dirty laundry. It's not that this stuff was private. It's just some of it's sort of back in the day. And as an important point, I want to make sure that people understand this is not that I think that FreeBSD is bad. It's that I think that FreeBSD has made some mistakes.
02:41
And to learn from mistakes, you really need to go back and understand why they happened. So going through that, and let's lay out what I think the troubles with FreeBSD actually are. FreeBSD is big. FreeBSD is old. And FreeBSD's leadership, historically, can be slow or hesitant to act.
03:03
And so let's dig through those a bit. FreeBSD is big. It's might not be the biggest code base out there. It may not even be the biggest open source code base out there, but you're not gonna get a full modern Unix OS in a small code footprint. And this is also just the subversion repository
03:23
that we have. These numbers are from January, I might add. The subversion repository takes up 3.1 gigabytes on disk. That's not a huge amount of data by modern standards, but it's a fair amount when you're the person who has to wade through it all. And I'll point out, this is just the source repository. Ports is, if anything, bigger.
03:41
Isn't that just head? No, that's the entire subversion repository. That's head checked out, 600 meg of source. And again, that's a lot to wrap your head around because that's everything. There are 71, just over 71,000 files in there. And that's a lot of non-blank lines of source code.
04:01
And that includes comments, but comments are important too. And why does the size of the project matter? Well, as I already said, it's very hard to wrap your head around all that as one person. There are projects out there where one person can comprehend the entirety of a project. It also means that this is very hard to automate.
04:22
Doing a full build test cycle in this takes an enormous amount of moving parts and it's just not fast. You look at the tools that projects like Rust use and they're relying on your cycle time in order minutes, whereas we've got probably a cycle time in order an hour if we're lucky.
04:42
And when you're gonna be running that on every single commit and you get many commits a day, that's a lot of machine time to actually test that. FreeBSD is comparatively old by the scheme of things. It dates back 23 years at least. And that's not including the time spent as the patch kit
05:01
or as a actual BSD project itself, the CSRG project. Why is this relevant? Well, unit testing source code is a comparatively modern thing when you're talking about that timescale. Back in the day, we didn't write unit tests and we certainly didn't design code to be unit tested.
05:20
Kernels are notoriously fun to unit test for various definitions of fun. But even the user land code is not necessarily designed to be easily tested. That's changing as we try to add tests to it, but it's still a challenge. And if you want to automate unit tests, you kind of have to have unit tests to automate. And our coverage is still not awesome.
05:43
And that then plays into another element later, which is that means we have to rely on the community to do code review and to understand the project and to understand what a change means. And when you get large projects landing in that especially touch lots of places or lots of kernel code, that can cause problems.
06:02
So first release of FreeBSD was in November, 1993. I actually emailed Kirk McKusick and asked him how things worked in the original BSD project. So they used SCCS for anyone who knows what that is. It was a proprietary AT&T tool.
06:21
Oh, here's Kirk, he can tell us all about it. They used SCCS, which is a proprietary AT&T tool. They had a mailing list for people to send bugs to. Distributions were done by mailing tapes. Patches in wood were generally done by email or via tape again.
06:40
And once remote logins were a thing, because they weren't always a thing, some trusted people given logins to the machine hosting the source code repository and were allowed to make changes. There was a core group of about four people managing the project back then. So moving on to my last point, FreeBSD's leadership can be slow or hesitant to act.
07:03
FreeBSD's leadership for the people in the room who don't know what it is, is a nine-person team elected from the ranks of active committers by the active committers. Active committers being defined as people who've committed something within the last 12 months. Core tends to be very consensus-driven, which is a good thing in terms of being able to stand by your decisions, but it also means that it can be slow
07:21
to come to a decision. And there are cases that come up where that lack of speed can cause more problems while you're trying to get the solution in place. Core tends to not want to take sides, especially in technical discussions.
07:42
Core has historically not wanted to be in control of architectural direction. We certainly don't want to be dictating it from core. And that flows back to a bunch of other things that have gone on in the past. So let's look at core team. FreeBSD's core team was originally
08:01
the four people who founded the project. So we're talking Jordan, Rob, and David Greenman, and I'm blanking on the fourth. Doesn't matter. As people contributed changes to the project and gained a level of trust, they'd be granted access to the system hosting the source code repository and the ability to commit their own changes.
08:20
This is very much similar to the way the CSRG project worked. And this is known as a commit bit. Committers who became especially well regarded were eventually invited to join core. And by invited, I mean often Shanghai. The problem that this led to core being perceived as somewhat of a cabal. Since core wasn't really answerable to anyone else.
08:41
And they were kind of the committers that were better than the rest of the committers. Along with this, a bunch of the earlier core team members were starting to burn out. And so around 2000, the elected core model was introduced. Just stepping away from that for a sec, it can be interesting to think about
09:00
what FreeBSD might have been like under a more benevolent dictator for lifestyle leadership like Python or Linux. I don't really think it matters whether you have a committee or a BDFL. Because if you look at the way Linux and Python actually work, they delegate authority in a lot of cases. And so the BDFL doesn't always deal
09:21
with the issues that come up. And I think it's also pretty fair to say that Guido van Rossum and Linus Tolvalds are fairly different people. So what I'm gonna do now is go through some of the things that have happened in the history of FreeBSD and talk about how they were dealt with by both the project and the leadership
09:41
and how maybe they could have been dealt with and can try and compare them to some of the other projects and how they've dealt with similar things. So tools and processes. When I first became involved in FreeBSD, it was using CVS. It was using IRC for a lot of sort of incidental communication
10:00
and the more official communication was done via mailing lists. And we had this wonderful tool called NATS for bug tracking. And why were these chosen? Well, basically at the time, there wasn't a lot of other choice there. The original Unix and BSD version crawl systems was SCCS.
10:21
It was proprietary and it wasn't really open source until a lot later. So with FreeBSD starting in around 1992, there weren't a lot of options. RCS shows up around 1982, but then CVS arrives as a bunch of shell scripts over that. You get other options showing up later in the piece.
10:40
But really back at the time, there wasn't really much else to choose from. So you might ask, given with things like BitKeeper showing up in 2000 and Subversion, why didn't we move to Subversion earlier than we did? Well, so one of the things that FreeBSD got into fairly on was the stable branch release model.
11:02
The exact mechanics of that have changed, but initially, if FreeBSD was a single stream of development, occasionally you'd sort of say, okay, stop everything, we're stabilizing, and you'd stabilize it and then ship a release. That gets unwieldy over time. And so the stable branch release model showed up.
11:21
This also tied in with the history of Walnut Creek and the CD-ROM delivery model that they were using. But one of the interesting things to note is that the only thing we used branches for in CVS was the release model. I believe NetBSD uses a branch in CVS, used or uses,
11:40
I don't know, is NetBSD still on CVS? NetBSD makes a lot more use of branches in CVS, but FreeBSD was very hesitant to do it because CVS branches are painful. CVS has a bunch of other problems too, being its commits are not atomic because it's a wrapper around a bunch of RCS files,
12:01
the changes happen to each file individually, and there's not a lot tying them together except maybe a timestamp, if you're lucky. So we're changing what changes were in which branch becomes a painful and it's just not fun. So eventually, and especially after the arrival of distributed control systems like Git and Mercurial, there was a lot of pressure to move
12:20
to a more modern version control system. Problem was, which one? Perforce had shown up in the FreeBSD project around 2000. Perforce is proprietary, but they liked us, so they gave us free licenses. Perforce has a bunch of advantages over CVS, commit atomicity, much cleaner branching model
12:42
and stuff like that, but it was proprietary, which was a turn off for some people, and also its interaction model can be somewhat odd if you're used to things like CVS. The other contenders that you can probably predict, Subversion was a big candidate for a long time, but Git, Mercurial, I think someone proposed Darks
13:01
at one point, so. But the real question is, how does a community agree on a change of a tool that's fundamental? Any project that goes on for a length of time is probably gonna have to deal with this question at some point. Version control system is the biggest one, but changing version control systems is not a small task.
13:23
It's an enormous extract, transform, load process, especially when you're dealing with something the size of FreeBSD's repository. So these things need to be carefully planned and tested, and so you need to know where you're going to go. This is a quote from Peter Whem.
13:41
Peter Whem is the guy who moved FreeBSD's version in 2008. There had been a lot of discussion on version control systems starting around 1999, advocating everyone you could think of. Some of these have progressed to a proof of concept conversion from CVS, and one of the more interesting things that was discovered during all this is that a lot of SCM conversion tools,
14:02
especially the open source ones, assume that you're dealing with your standard open source project size code base, which is not that big compared to ours. And so a lot of them believe that you can hold the entire repository state in memory and then dump it back out again, which is not gonna work with FreeBSD's.
14:22
And the other problem that we had was that our CVS repository had had many, many, many horrible things done to it over the years. The tamest of those is probably what people in FreeBSD of that time remember as a repo copy or a repo move where we would take the underlying RCS files and copy them to a new location
14:40
so that you'd end up with a new copy of those files that maintained the history of the files they came from. You're not supposed to do that, but it kind of works. But there's a lot of other stuff that we did to CVS that made it a much more interesting conversion job than I'm pretty sure any CVS conversion tool developer has ever tried.
15:00
So Peter, basically in typical Peter Whem fashion decided that he had the keys to the repository and he was just gonna do it. So he started working on getting a proof of concept conversion to subversion, which he picked because it was a fairly close match to the CVS model that we already had and both Git and Mercurial
15:22
and other tools could generally talk to it. And he kept doing conversions on it until he had something that he was comfortable with. And then he basically came up and tell everyone, right, I've done this and we're gonna do it now. So that took about nine years, eight or nine years
15:42
to get us from CVS to subversion. So with that in mind, a quick diversion via Python. You had a question? I'm just specifying four more years for the ports to be in the doc tree. And another four years for ports and docs. So a quick diversion via Python. Python also started in CVS as did everyone else at that time because there wasn't anything else.
16:02
And eventually a CVS was hosted on SourceForge. Since then, they moved there to subversion hosted on their own infrastructure in 2004, then to Mercurial on their own infrastructure in 2009 and then to GitHub in 2015 slash 16. All of these have been managed
16:21
using Python's Python enhancement process. So it's not, I don't think it's a goal of your project to change source code tools every time that a new thing comes along. But when the core amount of developers actually decide that they'd like to use something, it's good to be able to respond to that. So the Python enhancement process or PEP
16:43
is documented in PEP number one. It's very similar to the RFC process that you see out of the IETF. There's various other similar things in projects like your Lumos and Java. But the gist of PEP is that there's a group
17:00
of PEP editors who will help you write a PEP document that will document the change you want to make. And that change can be pretty much any kind of change you would see within a project. It could be code related. It could be saying we should be doing this kind of process in this way rather than that way. Or it could be a change in community structure or process.
17:22
When it's up to a reasonable standard, you submit it for consideration and either the Python benevolent dictator for life, AKA Guido van Rossum, or one of his chosen delegates will make, in consultation with other people, will make a decision as to whether it's accepted or not. So that gives you a really good vehicle
17:42
for progressing this kind of change within a community. Python isn't the only community with that one, but it really just, it allows a level of focus that you don't get when your entire discussion venue is mailing lists. So another example of people who've moved around is Django.
18:04
They moved from self-hosted subversion to GitHub. They don't have a process like PEP, but their team is a lot smaller. And so it's a lot easier to make those kind of changes. And this is another aspect of FreeBSD being big is that there's a lot more people and it's a lot harder to kind of manage them or get feedback from all of them in an effective manner.
18:22
It's like the difference between a small company and a large company. So how could we have done it better? Looking back through the email archives, a lot of the suggestions and pressure we move off CVS didn't result in workable solutions. There was nothing kind of focusing that effort into a common point.
18:42
So a lot of it was just lack of consensus, not enough of a group formed around any of the suggestions to actually progress it. And so really, in the absence of core actually stepping in and saying, okay, we've looked at all of the proposals and we think this one, we were really reliant on someone like Peter Wham to come along and actually change it
19:01
because he was the only person who basically said, damn the torpedoes and went and did it. Sorry, what was that? It's easier to ask for forgiveness. At some extent, yes, better to ask for forgiveness than beg permission. So having looked at tools and process stuff, let's look at the more fun stuff.
19:22
Community interactions. Specifically, we're gonna be looking at how a community deals with bad behavior. Small communities are nice in this regard because you kind of know everyone, everyone's kind of on the same page and if they're not, you feel comfortable enough standing up to them saying, that's not on, can you cut that out?
19:41
That's less doable when you're a project the size of FreeBSD. And so when I talk about community interactions and dealing with it, it's almost talking community HR. Human resources sometimes get a bad rap but they're a bit like project managers. When they're good, they're amazing and when they're bad, they're significantly less so.
20:01
When they're absent, things can go really, really wrong. So from at least the second core, which was the first elected one, there was a document called the Committer's Guide. It detailed the basics of how to participate in the project and it had two sections that covered interactions with other developers.
20:22
The first section was titled Developer Relations and contained advice for dealing with developers who are working in the same area as you. It ended with several paragraphs like this. Do not impugn the intentions of someone you disagree with. If they see a different solution to a problem than you or even a different problem, it is not because they are stupid, because they have questionable parentage or because they are trying to destroy your hard work,
20:42
personal image or free VSD, but simply because they have a different outlook on the world, different is good. And it goes on to say that you should disagree honestly. So argue your position from your merits, don't go, as we say sometimes in Australia, play the ball, play the, hang on, play the man, not the ball. No, play the ball, not the man.
21:02
But basically be open-minded about the discussions you're having. And then we also had the committers big list of rules that looked like this. Accept correction, we are all fallible. Ask for help, seek out peer review. And then respect other committers.
21:21
Oh, sorry, this is the big list of rules. Respect other committers, respect other contributors. I think that's a really important differentiation to make by the way, or rather respect all contributors should be number one. Discuss any significant change before committing, respect existing maintainers, disputed changes must be backed out, changes go to free VSD current before stable. Don't fight in public, no one likes that.
21:43
Respect all code freezes and so on and so forth. Following this list, there's a couple of paragraphs describing what core can do in the case of breaking these, which amounts to suspending or potentially removing commit access. Repo admins could also temporarily suspend access if someone was doing damage to the repo, so on and so forth.
22:00
Then there's just some discussion on the rules, and that's basically what the project had as what anyone would consider a code of conduct up until mid-2015. And as code of conducts go, it's not too bad, but it sort of, it doesn't keep pace as we'll see a bit later.
22:21
So having discussed that, let's talk about Matthew Dillon. Matthew Dillon joined the Free VSD project about 1994, and he's an extremely talented developer. I will not dispute that, and no one I've ever met has really said anything else. At the time though, a team player, he was not.
22:45
He had a bad habit of believing his changes were right and correct and committing them without regard to any of the maintainers of the area that he was working in, and generally without any regard for anyone else working in that area at all. I'm gonna qualify that by saying that I've not dug into his work after he's left Free VSD,
23:01
so he could be a very different person now. I am discussing the Matthew Dillon of the late 90s, early 2000s. So one of the major bits of work that was going on in Free VSD in about 2002 was fine-grain multiprocessing in the kernel. Free VSD three and four both had a giant kernel lock.
23:23
Multiple user space processes could be running in user space, but as soon as one of them went into the kernel, they took a lock and nothing else was allowed in there until it exited. That lets you have some level of parallelism, but it's not the best. Pretty much every kernel that you look at does this, including Linux, though I did it at the time.
23:41
Question becomes, how do you break that lock up and how do you allow more processes to be in the kernel at the same time, multithreading the kernel? One of the things I like about Free VSD is that it likes to think these things through a bit. One of the people that was doing a lot of that thinking at the time was John Baldwin. And another one who had a bunch of different ideas
24:02
was Matthew Dillon. John had come up with a design and he sort of documented the high-level version of it. He'd gotten feedback on it. He'd been researching the way BSD OS did it, the way Solaris did it, and he'd come up with a bunch of really good ideas. Matthew Dillon also had a bunch of ideas. I think his ideas mainly linked back to Amiga-DOS,
24:24
but it's just not as new. Anyway, but he had not really written them down and described them, and he certainly hadn't sort of solicited a whole bunch of feedback on it. And so with the fundamentally different ideas that they had,
24:42
there were already bits of argument going on about that. This came to a head over a thing called critical sections. Critical sections are a piece of code where you never want to be preempted. John's initial cut of how to do this just turned off interrupts on x86. This was a perfectly fine way to do it,
25:01
but it's, you know, x86, turning interrupts on and off and on can be a slowish process, and so Dillon didn't want to do that. He wanted to leave the interrupts enabled, but just have the low-level inter-corrupt code track that an interrupt had come in and then dispatch it when the critical section exited,
25:20
which is a very architecture-specific optimization. John objected to this, pointing out that he didn't want to optimize it until the API in question had been nailed down properly because it was still kind of in flux as to how these things would actually look. And Matt just committed his optimization,
25:44
which is not fun when you're in the middle of a discussion about how it should actually work, and it took a lot of discussion, and then eventually, he backed it out. Core was then stuck in the middle of a month-long debate between John and Matt over the change. John is, as John is, very patient about it,
26:04
and was obviously listening to Matt's input while standing by what he originally wanted to do in his plan. Matt, on the other hand, was less patient and less nice. Eventually, there was a compromise reached where some of Matt's optimization went in while maintaining the machine-independent,
26:22
machine-dependent split in a way that made John more comfortable. So looking at this, this, to me, is a rockstar ninja 10x cowboy developer problem. You've got one person who, while I undoubtedly have excellent technical chops,
26:40
is refusing to listen to anyone else and has decided that they are the most important person in the room. Digging back through the mail archives, I really found a lot of attempts to try and make Matt listen to reason and to John, who was also reasonable. And there was a fairly clear understanding,
27:01
at least from Core, who was in the middle of all this, that Matt was really approaching this in a way that wasn't helpful. The problem is how you deal with it, because Matt clearly wasn't going to accept being told to keep his changes out and do what, you know, work with John more. He pulled every status game in the book.
27:21
You know, you're holding back my development, you know, everything's moving on, we could be doing better, but you're not letting me do the things that will make it better. There were plenty of other areas in this work that he could have been looking at that weren't critical sections, but he really just decided to choose this hill to die on. And the problem in a project like FreeBSD
27:41
is because everyone's volunteers, you really can't tell them not to work somewhere or to just go away and do something else, because in a lot of cases, if you do that, they'll just leave. So there was a refrain that has persisted through the years and is now on the back of several T-shirts around here. It was somewhat derisive at the time,
28:00
and you know, what is Core going to do about it? Matt was not the only offender in the project in that sort of thing. The project had something like 300 committers, not all of them active, so you're gonna have some people butt heads. Core was fielding a lot of complaints about bad behavior at the time, and some committers ended up resigning over constant infighting. So Core set about trying to rein it in.
28:21
So the following extra rules were sent out via email, though never put in the committers guide. So this is trying to clarify the kind of suspension rules that are gonna kick in if you do the wrong thing, which is basically, if you commit to stable branches during code freeze, you'll get suspended.
28:42
Committing to security branch without approval from security officer will be suspended. Commit wars are bad. So you know, again, this sort of builds on the kind of code of conduct stuff that was already there, and kind of tries to nail down the rules a bit more.
29:03
They feel a bit authoritarian, and there was some discussion about it on the developers mailing list on the for and against side. It first got put into practice on someone who wasn't Dylan, but who had committed to stable without committing to current first. Dylan's first taste of it came when he committed some changes to the dynamic linker code without going through the maintainer.
29:22
The problem was that the crime he was being punished for there wasn't the crime he really needed to be punished for. The problem with someone like that, with Dylan, which we're better at spotting these days, I think, is that it wasn't always one thing. It was a lot of small things that built up
29:41
until he do something that caused a reaction. In this case, the infraction was minor, but I get the impression from reading through the archives that Core jumped on it to try and make a point. And so things continued on after that, and I know that I recall a lot during that time that developer interactions were quite fractious. But again, Dylan wasn't the only offender.
30:02
We also had such quiet and retiring people like Paul Henning-Camp and people like that who also ran afoul of this kind of stuff. But Dylan kept on alienating people rather spectacularly until things finally got a bit much. This is the commit log that began
30:21
the end of Dylan's FreeBSD career. The change was mildly controversial because it changed a bunch of things that could be seen as a layering violation, as introducing a layering violation. And on top of that, it came in the freeze window just before the release of FreeBSD 5, which was towards the end of a lot of the first stage
30:42
of the really painful work to getting fine-grained SMP in the kernel, touching exposed API and ABI stuff at that window could really cause problems with the release process. And so the release engineer essentially said, back this out, and he did, with this commit message.
31:01
This is the commit that got Dylan's commit bit revoked. As you can understand, that kind of tone in a commit message doesn't go down well. His response to being asked to tone it down pretty much sealed the deal, and his commit bit was taken away. And at that point, he took FreeBSD 4.9-ish
31:22
and started DragonflyBSD. But what this comes down to, then the problem with this dragging on as long as it did is leadership is hard, and volunteer community HR is hard too, especially when it's being done by volunteers.
31:41
I'm almost certain that FreeBSD isn't the only project that's had a situation like this. But I think the only thing that we can really criticize is that this pattern of behavior wasn't dealt with earlier but that would have taken, again,
32:03
a much more proactive stance on the part of core, which is not really what core is known for. Under a more modern code of conduct, John might have been able to report Dylan for some of the actions that he'd taken, and that might have changed things in a bit, but the biggest thing you can take away from this
32:22
is that community volunteers, you have a lot less control over the people in your project than you would in an employer-manager-employee relationship. And thankfully, after this, removing people's commit bits remains very rare, and things were a lot quieter after that for a while. To be fair, with core, at the time,
32:41
it was not, the fact that most of the communication between people is in the electronic form is some kind of barrier, because some people just can't deal with conflict or different opinions through mail. And to be fair, with core, at the time, it was during, I think it was the first BSD card,
33:03
and we tried, and core tried to deal with Max in person, but that didn't help. So yeah, just for the sake of the people online and stuff like that, Olivier brings up a very valid point that a lot of this discussion happens via forums like email
33:20
with a very geographically distributed group of people, and that acts as its own barrier to reaching resolution. And secondly, that a bunch of this stuff was tried to be dealt with at the first BSDCon in person with Matt, and that that didn't help either. So yeah, things quieted down a bit after that until this thing called gamergate showed up.
33:44
A candidate for the most obvious and least controversial statement in history, gamergate was a burning trash pile made of smaller burning trash piles. I'm not gonna go into what gamergate
34:00
actually was too much. If you want the long version, you can come and find me outside and I'll use as many expletives as you like. But it kicked off in the gaming community somewhat and rapidly became something it was really hard to avoid on Twitter. But the connection between gamergate and FreeBSD
34:21
comes through the far less controversial figure of Randy Harper. So another sort of not very controversial statement, Randy can be a pretty difficult person to deal with. Her history with FreeBSD is that she was a sysadmin. She started making some fixes to the FreeBSD installer,
34:41
which always needs help. The person who was committing these changes on her behalf proposed her for a commit bit, which is how these things worked. She was granted a commit bit. She did a bunch of work on sysinstall, which like I said, everyone needs to work on the FreeBSD installer. But then she got a job at Amazon AWS, which wasn't FreeBSD focused and she didn't have a lot of time to work on it anymore,
35:02
but she was still hanging around a lot of the community. Fast forward to the aforementioned burning trash pile of gamergate. Randy was working at a gaming company doing some DevOps work. She, as she is wont to do, started getting into the face of various gamergaters.
35:21
And as they are wont to do, they started getting in her face. But the way they get in your face is by outright mobbing you. And it's a very unpleasant experience to have. So one night, while drunk, she decided to try and get rid of them with a Perl script, and she did. She wrote a script called gtautoblocker
35:41
and it sort of analyzed the social graph on Twitter, tried to work out whether someone in particular was a gamergate, and if they decided that it wasn't, it just preemptively blocked them. So she could no longer see gamergate. And the best way to make a gamergater lose their mind is to make it so you can't hear them anymore.
36:01
And this worked. And it made the gators lose their utter goddamn mind. And so this, to them, is a challenge. And so they ratcheted up the harassment campaign and started targeting everything they could find to do with her, including free BSD. So CORE started receiving emails asking them to do something about Randy Harper
36:22
in late 2014. They were fairly easily rebuffed because what she was doing was clearly not free BSD related. Randy would also drop emails saying, these people are about to start making trouble. And so we could be forewarned, for instance, people started trying to pretend to be her and getting into some of the private project IRC channels
36:42
so on and so forth, all kinds of fun. This was all fine until someone who was within the free BSD project decided to start taking shots at her on Twitter. And then this escalated onto one of the aforementioned private IRC channels. This person then leaked a bunch of logs from the channel,
37:03
which is bad in and of itself, but they also leaked them to a person named Milo Yiannopoulos who was with Breitbart at the time. And if anyone has not heard of him, keep it that way. He's a terrible human being.
37:24
So at that point, Core kind of had to get involved. Randy had sent an email to Core detailing a list of things that she wanted to see happen. One was the creation of a more formal code of conduct. The second was a public statement
37:41
saying that we were investigating this behavior and that we didn't stand for the harassment involved and that we were committed to inclusion and so on and so forth. And the last was this person get kicked out of the project. Core responded that the COC was in progress, but sanctioning the developer was tricky due to where things had happened. Leaking of private IRC logs was a clear breach
38:02
of documented rules and that was perfectly something that Core could deal with. But Core was very unsure as to what to do about the abuse on social media. I know that at least one member was offered just booting both Randy and the person in question because they were all just making trouble. And which just highlights another problem
38:22
is that a bunch of people on Core didn't actually realize the extent of what was going on. Social media dogpiling is not something that's immediately obvious when you're not on that social media network or seeing what's going on. It's really hard to understand what Twitter dogpiling looks like and feels like when you've never seen it.
38:44
And so, you know, a bunch of people on Core at that point hadn't even heard of Gamergate. And, but after that, they had to decide whether or not they could kick someone out of the project based on what they did in a venue that wasn't under project control.
39:01
In the end, the developer in question, probably realizing what was coming, handed in their commit bit voluntarily before Core got around to working out what to do with them. But that wasn't the end of it. And the fact that it took Core a while to spin up on understanding that, it's not exactly an example of Core being hesitant, but it's just another example of the problems
39:21
that you have in a volunteer project when things happen that you don't necessarily understand at first. And it also was compounded by the fact that the person was still attacking Randy on Twitter, Reddit, Facebook, anything they could get their hands on. And it's not, this isn't quite the end of the story yet.
39:43
Core made good on the code of conduct promise, but the original code of conduct was very much a product of an old school FreeBSD or understanding of what a code of conduct needed to look like. And the COC we have now is a patched up version of that one. But things like the original one
40:01
included the word meritocracy. Meritocracy is an interesting word because on the face of it, it seems perfectly fine. We are a project that recognizes people based on merit. But the problem is that merit is subjective. Merit can mean a whole bunch of different things. And if you're from a group who has a systematic bias against them,
40:24
so if you are not the kind of person who's gonna find it easy to get involved in a project in the first place, it's very hard to show merit and thus be recognized. We have a new code of conduct that will be published very soon. This one's been developed in consultation with other people.
40:41
I'm much more happy with it. And yeah, watch this space. The fallout from all of this was at least in some ways fairly spectacular. Randi was not at all impressed with how we'd handled it, which is fair enough. She published a blog post at the end of 2015 that is rather scathing.
41:05
The thing with Randi is that she can be very hard to deal with, she's blunt as hell. But especially through Gamergate, there are a lot of ways in which she was not wrong. And you can't just write someone off or their message off because you don't like the way it was delivered.
41:22
The heat that she gave us over that, I think, was largely warranted. I'm hoping that we can make a bunch of things right with the new code of conduct and with a bunch of other things. And it was things like this that were part of why I decided to run for core in the first place.
41:42
So building on all of that, how do we fix the troubles with FreeBSD? Having looked at some of the nastier bits. We can't really fix the fact that FreeBSD is big. The only way we could do that would be pulling an x.org and splitting everything up. And I don't think that that's gonna happen
42:01
and I don't even think it's necessarily the right idea. There are certainly some things we could remove. I think we could have an argument as to whether send mail belongs in base, but let's not do that right now. But even just packaging base, I think, would help. And there's a lot of work in terms of improving our incremental builds
42:22
and stuff like that, which can all help speed up the cycle time if we want to improve our sort of build test cycle speed which is mainly where the bigness becomes a problem. Then the other one is just getting more people in who can take over and understand various subsystems so that we have the people necessary to review changes that come in.
42:41
And in that case, in that space, you could help us. If there's anyone in here who's not currently involved in FreeBSD but is interested in it, come talk to me and I'll try and help you get into the project and I'll find you something to do. We also can't really fix the problem that FreeBSD is old. But that said, we can start restructuring the code
43:03
in the system to be more amenable to testing. And we can start looking at some of the community attitudes that might be holding us back and try and work our way around those. And we can start looking at our tools and processes to try and make them more effective and useful. And then the last one is leadership.
43:23
When I look at projects like Python that can sort of do four version control migrations in the time it takes us to do one, it sort of makes me go, I wish we had the ability to do that. I feel that we're making the right moves
43:40
and I fully agree with Olivia's point that the fact that we're a geographically distributed team makes it slow to get to some decisions. But I think with things that we're bringing in like the FreeBSD community proposal process and stuff like that, I'm hoping that we can start trying to get leadership
44:02
to do the right kinds of decision-making. I don't want core to become a technical review board, but I do want it to assist in the process of moving larger technical change forward. Because the real problem with FreeBSD is that it's made by and led by volunteers.
44:23
There are very, very few people, if any, whose job title is solely make FreeBSD more awesome. There are a lot of people whose job title is make FreeBSD more awesome for my employer or the person who's paying my bills right now, but that can be a very different thing. It's still valuable, but there's nobody out there who just gets paid
44:41
to sit there and do whatever they want on FreeBSD all day. And core is a set of volunteers derived from a set of volunteers. Nobody on core gets paid to be on core. And to stay alive and stay relevant, we need to make sure that all the volunteers that we have want to stay,
45:02
which requires the leadership to set and enforce good rules to make sure that good people have a good time. And it also requires, I think, that the core team gets out of the idea that it should never make a technical decision. I think with the FCP proposal,
45:24
core does actually take a role in that space. But the role that it's taking is trying to crystallize the opinion of the project as a whole at the time. And so it's an interpretive decision, much more than it is an actual sort of dictative decision,
45:40
if that makes sense. Those are not in the room, what's FCP? The FreeBSD community proposal process. And so I think if we can just make those tweaks to the project and try and get the friction out of what we're doing and generally just make FreeBSD
46:01
a better place to be for the volunteers that make it up, then I think FreeBSD is going to stay relevant and stay alive for a very good long time to come. So with that, I would like to thank a few people. Kirk McKusick for answering my email on how the original BSD projects functioned. Jordan Hubbard for giving me some background
46:20
on the early days of FreeBSD. Russell Keith-McGee from the Django project for giving me some answers about how Django have dealt with things. And Nick Coughlan from Python for similar background. And with that, thank you very much.
46:42
Yes, George. So, big high stopper model, also known as fascism. Yep. But I noticed that the two fascist languages, Python and Perl, have failed to go between versions, even though they can change version control every week.
47:01
But Python 3 and Perl 6. I think you're wrong about Python 3. Its uptake is increasing a lot, as of about 3.5, 3.6. Right, but my point is more about leadership. I mean, he made those changes a long time ago. Yeah. So I think even if core were a technical review committee
47:20
or were a bunch of people. I think there's a lot of interesting discussion you can make about Python 3 and Perl 6. I think they're both very different things. I agree with you that Python 3 has taken a long time to bed down. But if you look at the stuff that was coming out around the time when it was still called Python 3000,
47:41
they expected it to take a long time. I don't think they expected it to take this long. But they did expect it to take a while. But the thing is that that has kind of gone through now. And there's a lot of now Python 3 only libraries out there. There's a lot of Python 3 work going on. Heroku is now defaulting to Python 3.6.1.
48:04
Instagram. Instagram, yeah. So I take your point. And that was what I was trying to get at. So I don't think BDFL versus committee really makes a difference. It's more the way in which that leadership operates. And like I said, Guido is a very different person to Linus.
48:22
And so the vibe inside Linux is very different to the vibe inside Python, making absolutely no value judgments on those two in either way. Well, it's my life, and it's a piece of human being and the other one is mine. Yes. In the same vein as the other open source projects,
49:00
there's only corporate resources for whether it
49:11
be legal matters or whether it be HR related aspects, et cetera.
49:25
Yep.
49:40
It's not that I think that FreeBSD is a singular corporate entity backing it. I mean, we have the FreeBSD Foundation, which can deal with some of the legals and things like that. But I'm not saying that there needs to be a red hat for FreeBSD or a canonical for FreeBSD or even like a larger thing, like Swift gets run out
50:02
of Apple, but the people who work on Swift have a fair amount of leeway in how they actually put it together. My point was more that FreeBSD, to get a job being paid to work on FreeBSD is rare-ish. To get a job where you get to work on whatever you think
50:21
needs to be done in FreeBSD as opposed to what your employer wants done on FreeBSD is pretty much not there. I think the only person who could claim to have that and even then he needs to answer to Ed is Kib, Kostek. So yeah, I think it's just, it would
50:43
be nice if there was more money around for sort of generalist improvements to FreeBSD. I don't know how that happens without finding a corporate backer, but it would be nice if we could thread that needle.
51:04
Lawyers and HR side of things, if you go to athletics, there's all these screw ups where Suze or Red Hat pushed something in the middle, and even though they have the money, it didn't end up going anywhere. ACIO, virtualization, all these things
51:21
between them, even though the big money was pushing them. So even there, the technical side of things was independent of the money, but what they do have is lawyers looking up and licensing and getting the code conduct right, HR, a real HR thing.
51:40
Well, one of the things that, sorry, the statement was around using available corporate money to do things that the programmers are less good at, like legals and HR and stuff like that. Just one thing I'll remark on regarding that is that for the new code of conduct, we did actually get outside consultation in on that, and the foundation generously paid for that.
52:02
Given that good HR is a skill set and one that not all of us have, what could be done maybe to get volunteers with that skill set interested in helping the product? That's a very interesting question, and I'm not really sure how to do that. It would be interesting to see if there were some people
52:22
with an HR background who were willing to put their time in, but I think that HR for a volunteer group is a very different thing to HR for a corporate environment. Your level of ability to apply sanctions is greatly reduced. So I'd just like to illustrate what the Gamergate dog-piling can look like.
52:44
In 2015, Garrett Bistec and Randy Harper gave a presentation, and I posted on Twitter, but I enjoyed it. That's the word I heard earlier this year. I mean, the discussion on Reddit, somebody doesn't like what I say, and they go back through my Twitter history and bring up that tweet, and they explicitly say that,
53:03
oh, I'm supporting Randy Harper in this, and trying to use that as a way to attack me. So they went through my, I probably posted 15,000 times since that tweet. I keep being blocked. They went through my history to go find that. And I was just, one comment,
53:21
because they decided they didn't like me. Anything else? More up there? Do you have a proposal?
53:41
Well, okay. In terms of community process, the big thing that I'm working on at the moment is the FreeBSD community proposal setup. Describe that for people who haven't heard it. Okay, so, FreeBSD community proposal is derived from things that I described in there, like Python enhancement proposals, also RFCs,
54:02
Illumos' RFD, is it? Yeah. Yeah, and the general idea is that any contributor to the project can propose a change to the project by describing it and getting it discussed by the community and then voted on by core.
54:21
So if we want, and changes can apply to any part of the project. It can be technical. So you could propose a change to an API that could be more drastic than we would normally do. You could argue for a change in tooling. So you could say, we need to move from Bugzilla to whatever is better than Bugzilla.
54:42
Jira, we're gonna move to Jira. We could hurt you. You could propose that. I'm gonna say subversion to Git. Subversion to Git is another example. Subversion to GitHub is another example. All of these things could be proposed. That doesn't necessarily mean they'll happen.
55:01
But what it does mean is if you do want it to happen, you need to write it up well. And you need to describe how you're mitigating any risks, how you're gonna be testing the architectural changes you're proposing. All of these things, they can be done. They can be discussed. They can be debated. But unlike previous discussions between certain common rational individuals like Alfred and Paul Henning,
55:21
we can actually take the discussion and crystallize it into a document so that we can understand the arguments that were made for and against a proposal and then have that as a record so that the next time this discussion comes around, we can point to the document and say, that's why we made the decision we did. And so what I'm hoping is that this will allow,
55:42
it's not meant to destroy bike sheds, but what it is meant to do is make sure that bike sheds, when they happen, are less likely to happen again. So that's the main thing I'm working towards in terms of community architecture, if you like. In terms of technical direction,
56:01
in terms of trying to reduce cycle time, I think there's a bunch of things that are in the process at the moment that are working. There's Lee Wen's doing an excellent job at trying to set up CI infrastructure for the project. Brian Drury's been doing excellent work on incremental build work. There's various projects in train to package base,
56:21
and I'm hoping at least one of them comes off. So yeah, I think there's a lot of things to look forward to in that space. One more question? Nope. Thank you very much.