Adventures in building open source software (part 1 of 2)
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 41 | |
| Author | ||
| License | CC Attribution - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/18683 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
4
8
13
14
16
22
25
33
39
00:00
Open sourceProjective planeJSONXMLUML
00:17
Projective planeSoftwareWireless LANJSON
00:36
Metropolitan area networkMaxima and minimaWeb pageForm (programming)FreewareInternetworkingComputer animation
01:20
Interface (computing)CoroutineSystem callSet (mathematics)Daylight saving timeOperations researchSystem programmingBuildingDirection (geometry)Source codeCodeMultiplication signMathematicsPhysical systemSoftware testingComputer fileSoftware developerBitConnectivity (graph theory)Functional (mathematics)Interface (computing)EmailSystem callOpen sourceUniform resource locatorModal logicComputing platformString (computer science)Wrapper (data mining)Virtual machineOperating systemProjective planeFluid staticsSoftwareRevision controlCompilation albumHypermediaPressureLaptopJava appletParameter (computer programming)PowerPCComputer hardwareCodeElectronic mailing listLine (geometry)Total S.A.Daylight saving timeProgrammer (hardware)Portable communications deviceOpen setFreewareCuboidProcess (computing)Data structureTask (computing)Message passingComputer animation
08:04
Level (video gaming)MultimediaTotal S.A.Power (physics)Binary fileOvalPhysical systemData typeType theoryCore dumpOpen sourceDemosceneUniform resource locatorQuicksortSource codeMessage passingRevision controlChainLibrary (computing)NumberComputing platformBinary codeBuildingPhysical systemComputer fileMultiplication signOperating systemData managementSet (mathematics)Dependent and independent variablesModule (mathematics)DatabaseGroup actionSpacetimeNetwork topologyConnectivity (graph theory)Projective planeFluid staticsCheat <Computerspiel>Fiber bundleHacker (term)Parameter (computer programming)Linker (computing)ResultantVirtual machineEmailInternet service providerCodeSoftware developerCompilerDifferenz <Mathematik>Source codeJSON
14:48
Link (knot theory)CodeProcess (computing)MathematicsBootingPhysical systemParallel portSource codeDirectory serviceComputing platformExecution unitMultiplication signOpen sourceFile systemSpacetimeSet (mathematics)MehrplatzsystemNeuroinformatikComputer fileDifferenz <Mathematik>DemonJSON
Transcript: English(auto-generated)
00:00
Hello, everyone. My name is Sivan Janyan. I'm going to talk about some of the things I've been working on over the last year with various open source projects, predominantly package source from NetBSD.
00:21
So to start off with, I started working on a project called KuvaChili, which is a captive portal software for use on wireless networks. What that is is typically when you go to a cafe or a Starbucks
00:41
and you connect to their public Wi-Fi, you're usually presented with some form of a web page which you authenticate, and then you're granted internet access. KuvaChili allows you to do that. And so I originally ported this to FreeBSD many years ago,
01:01
but I left it to grow stagnant. And as FreeBSD moved to Clang in FreeBSD 10, the package was broken. So I set out to fix the situation.
01:22
And it wasn't actually any restriction. Well, actually, there was only one restriction imposed by Clang itself. Everything else was basically bad coding habits within the code base, which Clang kind of raised to clean up.
01:42
The restriction that Clang imposed was that Clang didn't support nested functions. So I had to break functions out. And so everything would be called out logically rather than having one ginormous function with functions within itself, which would get called.
02:02
Everything else after that was basically defining the parameters for the function calls rather than just having an empty parameter list. And through that, I got things going.
02:20
After that, it was basically using deprecated functionality for setting up the network interfaces, which FreeBSD finally ripped out in version 10 that had been marked deprecated back in sometime in 2000, 2001.
02:40
But actually, this is a good thing, because the original method was it would take three or four calls to actually set up an interface, whereas the new method, it's a single call with a struct, and your interface is up and going. The problem is that the code base also
03:01
wasn't really written with portability in mind. So there's lots of horrible if defs all over the place. And I'm sad to say I had to contribute to this a little bit in that for Linux, everything is still in the old method. But if you're FreeBSD, OpenBSD, or Mac OS,
03:22
you use the new improve method, and off we go. So after that, I started looking into the source code. So projects like Covarity and also Clang offer a static code analyzer.
03:42
Running that against the code base raises lots of issues. And also, within it, rather than using the methods that the operating system provides, the developer opted to create his own methods instead.
04:01
So one of the examples is these functions that are prefixed with safe, which I don't know if they are safe at all. But instead of having something like strl copy, which is found on the BSDs and Mac OS,
04:20
he decided to create his own version, which is just a wrapper around the insecure string copy function. So this was a bit more painful. So I'm not actually originally a programmer, but the code base is heavily oriented around using
04:43
OTO tools and libtool. So what I wanted to do was basically rip these safe functions out, replace them with what's there in our libc, and then get the header files imported on platforms that
05:05
wouldn't have this in place. So if you're building on Linux, which doesn't have this strl copy, the necessary header file would be pulled in. So traditionally, if you weren't using the OTO tools and stuff like that, you would have your file in some common location.
05:22
And in your source code, you would put pound include and the header file to import. And I thought this would be the method that you would do for OTO tools. But actually, you do not want to do anything like that. And you focus purely on OTO tools and have tests written in OTO tools.
05:44
So when you're actually building it, OTO tools would probe your operating system. And if it doesn't find the necessary components, it itself would pull the necessary components in. So I lost quite a bit of time due to this.
06:02
But thanks to Darren Chandler from the OpenBSD project, he kind of prodded me in the right direction to doing things within AutoMake. And it's literally, I should actually have this up on the thing. It was literally a three-line check for doing the test.
06:23
So after that, these changes are still waiting to go in. There's a pull request on GitHub. And there's more to come for this. And after, I moved on to dealing with package source.
06:44
So on package source, I started off trying to build package source on old Mac OS, specifically Mac OS Tiger on PowerPC. And this old hardware is kind of very slow
07:06
and not very powerful. And I had an old laptop, which I was trying to do these builds on and trying to build things like GCC and stuff like that was taking a considerable amount of time and putting this laptop that I had under lots of pressure.
07:26
So I was given a Mac mini to kind of dedicate to the task and free this poor laptop from tortures of building GCC, which would take 48 hours if you wanted Java support and things like this.
07:44
And British summertime, even though this is a fairly small box, it actually still puts out quite a bit of heat trying to get things compiling. So I got on social media and asked around. And I ended up landing a machine
08:02
in MIT, which sits there. And I can just run builds. And it doesn't matter. And so this, what is up here, is the result of the first build of the entire tree on this machine at MIT.
08:22
And at the time, there was 15,000 possible packages that could be built on OS X Tiger. There was 8,500 built successfully. And then these were the key packages that caused the most breakages.
08:41
So for Mesolib, the reason that wouldn't build was because the tool chain that Apple provided wasn't able to generate the shared libraries that Mesolib wanted to generate. So originally, I thought most of that stuff is open source.
09:04
So I'll try and update the components myself and bundle that up as a separate package which people could use on an old version of Mac OS. The problem was that though the source code for what they call the CC tools is available,
09:24
when the Hackintosh scene came about, when Apple decided to move to Intel Macs, Apple withdrew most of the build documentation and components for building the operating system yourself.
09:43
So pretty much though you have the source code for some of these components, you don't actually know how to build it because all the documentation has been pulled. So I kind of cheated and built a static version of Mesolib
10:00
to kind of get around the issue of not being able to generate shared libraries with the tool chain that Apple had provided. For CMake, CMake was an issue where the build that the project provides for building
10:24
on Mac OS assumes that you're going to be building in a set location. And any other path, it cannot handle. So what would happen would you would build in CMake from package source. Package source needs OpenSSL, Zlib,
10:45
and some of these common core components. So that would be passed to the build system. And the build system would start building. And then it would get to the linking stage. And it'd go, oh, I know all of the components that I need are in the location that Apple provides
11:01
in slash developer. And that would blow up because it would be pulling in headers from a modern version of components and trying to link to 15-year-old version of components in another location. And that was quite painful.
11:21
For the various versions of Ruby, I'll come back to for the QT, that was not really solved. QT is a really big code base. And even the smallest problem, the diffs are quite big.
11:49
And because it takes such a long time, I kind of didn't pursue trying to fix this. Things that aren't here was the version
12:00
of GCC that was on Mac OS Tiger was version 4.4. So I wanted to bring in new versions of GCC so people would have a modern compiler. For most of that, really, it would build. But the problems were in package source where when you're passing parameters to your linker,
12:24
the version of the linker in Mac OS cannot handle having spaces in the paths that are passed in, whereas modern linkers can. So obviously, this hadn't been backported to older versions. So really, to get things going,
12:41
I just had to remove a space from a path and things would just build. So Ruby, turns out that Ruby has a very, very decent set of documentation for their API.
13:02
However, for the build documentation, it is absolutely non-existent. So when you go on and ask people, how do I build Ruby, it's quite a common response to be actually pointed to a package manager for your operating system and told, go and use brew or RPM or yum or something like this,
13:24
which doesn't really help you if you're trying to build it yourself. And literally, the problem was that the database module in Ruby was unable to cope with Berkeley DB in Mac OS because on Mac OS, the files are split
13:42
into two separate files. Whereas, Ruby expects it to be one. And that was it. There was one comment in the source code buried somewhere in the database module. And I lost a lot of time for that, and it's very annoying.
14:10
So by the new year, I've managed to actually exceed the number of packages available in package source from the Intel binaries.
14:24
So after that, I started looking around at new platforms to play around with on package source. And every geek kind of has a soft spot for BOS, right? So Haiku is still going.
14:42
So I thought I'd kind of play around with that. But with Haiku, that was really painful. And I didn't actually get very far because I'm not sure if this is applicable to BOS.
15:04
But with Haiku, there is no notion of multi-user system. It's a single user system, and you're automatically what is the super user. And for their file system, there is no file system as such. You have packages which contain snippets of file system.
15:25
And there's a daemon that starts up when the system boots. And this daemon takes your packages and union mounts all the packages that you have to form what appears to be a user land. And then you have a piece of writable space
15:43
on disk, which is your home directory. So for package source to work, you would be basically bootstrapping in your home directory, and it was a lot of work to actually trying to integrate it into the system. And the other problem was that Perl,
16:06
which gets pulled into a build quite earlier on in the process, would not build. Or it would build, but it wouldn't link. And the guys have their own package managing system.
16:23
But the way that they've actually implemented all their changes, rather than trying to integrate with what's actually there, they've just gone in and started deleting stuff and replacing it. So when you're looking at their changes, you have these quite extensive diffs
16:41
that you need to unpick. Whereas for us in package source, we don't touch the Perl code base. We just, at build time, we pass a settings file to say, build Perl with these settings. And the build goes off and does its thing, rather than actually modifying the source code
17:02
to hard code the settings we want to build with. And so I kind of gave up on this and thought about another platform to apply my skills at.