Keeping Current
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 24 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/15344 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Year | 2014 | |
| Production Place | Ottawa, Canada |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
1
5
7
9
12
13
14
15
17
18
19
21
23
00:00
Coma BerenicesData storage deviceParameter (computer programming)Kernel (computing)Stack (abstract data type)MathematicsInstallable File SystemServer (computing)MathematicsMultiplication signTerm (mathematics)Gene clusterOSI modelSoftwareBootingVirtual machineRevision controlDifferential (mechanical device)MereologyEnterprise architectureValidity (statistics)Point (geometry)Suite (music)MultiplicationCASE <Informatik>Computer fileRight angleDivision (mathematics)Projective planeStructural loadScalabilityConstraint (mathematics)Branch (computer science)Data storage devicePhysical systemStability theoryQuicksortTable (information)ChainInstance (computer science)Communications protocolLevel (video gaming)File systemEndliche ModelltheorieCircleUtility softwareRule of inferenceGame theoryGame controllerExpected valueCodeSoftware bugComplex (psychology)iSCSIComputer virusBitKernel (computing)Product (business)Block (periodic table)Differenz <Mathematik>Hacker (term)Pattern languageFront and back endsScaling (geometry)Parameter (computer programming)Adaptive behaviorLimit (category theory)Computer animation
07:34
Scheduling (computing)MathematicsMultiplication signStability theory
08:19
MereologyCodeSoftware bugDirection (geometry)MathematicsSource codeSoftware testingDisk read-and-write headScripting languagePlanningSoftware testingMathematicsIterationRevision controlMusical ensembleProjective planeGame theoryTime zoneBranch (computer science)MereologyPoint (geometry)Network topologySelf-organizationCASE <Informatik>Endliche ModelltheorieMetropolitan area networkMultiplication signGroup actionIntegrated development environmentDisk read-and-write headData miningPhysical systemEvent horizonFile systemStability theory1 (number)Game controllerScripting languageDivision (mathematics)Rule of inferenceValidity (statistics)Table (information)QuicksortRow (database)Image resolutionPhysical lawComputer architectureTrailMultiplicationCodeControl flowFile formatSocial classBitInformationGoodness of fitKernel (computing)Source codeParameter (computer programming)Software bugProduct (business)Computing platformCategory of beingSimilarity (geometry)Suite (music)Repository (publishing)Ocean currentPerfect groupProcess (computing)Local ringView (database)Operating systemUsabilityMessage passingAdaptive behaviorAddressing modeDifferent (Kate Ryan album)Front and back endsUML
17:49
Data management1 (number)MereologyLevel (video gaming)Cartesian coordinate systemVideo gameWritingBitWeb 2.0Computer animation
18:23
MereologySource codeSoftware testingCodeStress (mechanics)Disk read-and-write headComa BerenicesGame theoryMultiplication signInsertion lossMereologySoftware testingRevision controlQuicksortLine (geometry)Stability theoryCASE <Informatik>Position operatorValidity (statistics)Observational studyCycle (graph theory)BitMachine visionPoint (geometry)DialectPlanningProjective planeClosed setDecision theoryWhiteboardWordPhysical lawDisk read-and-write headQueue (abstract data type)Medical imagingLevel (video gaming)Branch (computer science)MathematicsEvent horizonTotal S.A.Software developerDressing (medical)Data managementDependent and independent variablesRight angleState of matterPhase transitionComputer fontPhysical systemTrailIterationTime zoneCodeGame controllerProcess (computing)Local ringStrategy gameFigurate numberNeuroinformatikComputer fileOnline helpMessage passingNetwork topologyInstance (computer science)Data storage deviceTesselationStress (mechanics)ResultantElement (mathematics)Adaptive behaviorDivision (mathematics)Order (biology)Proof theoryVirtual machineElectronic visual displayAxiom of choiceMusical ensembleDifferent (Kate Ryan album)Core dumpForm (programming)Web 2.0Software bugArithmetic progressionFlow separationSuite (music)Ocean currentEnterprise architectureTerm (mathematics)Drop (liquid)Conditional-access moduleDifferenz <Mathematik>Front and back endsAddress spaceBit ratePatch (Unix)Goodness of fitSource codeWave packetGeometryShift operatorLatent heatComputer animation
Transcript: English(auto-generated)
00:02
Hello. Hi. Haven, shush. Right, hi, my name's Benno Rice, I work for Icelon, and yeah, I'm going to be talking about, well this is kind of a, it's a story about how we
00:29
maintain our sort of internal fork of 3DSD and the joys and wonders involved. So yeah, so I work at Icelon, or rather EMC, Icelon. Icelon's product, for those of
00:47
you who haven't come across it before, it's clustered, scale out, network attached, storage. So working that backward, it's storage, so you put your data on it, you put your files on it. It's network attached, so instead of
01:03
most of the protocols being block based over something like SCSI, the protocols are file based over the network. We do support iSCSI and stuff like that, but most of the use cases we use, we have HDFS, we have NFS, we have SAMBA, that kind of stuff. It's scale out rather than scale up, which
01:22
means that instead of taking out smaller things and putting in bigger things, we instead get you to add more things, and it's clustered. So all our things get connected together by a back end fabric, and they all work together. The things themselves, we call them nodes. There's not a
01:41
whole lot that's special about them. They're fairly off the shelf x86-64 kit. We do a lot of internal qualification on them. There's a few custom bits, but not a whole lot. The software, however, is a lot more interesting. The software we have is called 1FS. It's a modified version of
02:06
FreeBSD, which is why we're here. It's a fairly significant modification of FreeBSD. We do a lot of changes from just tuning parameters. We tweak some of
02:20
kernel logging stuff. We have a larger kernel stack. The smaller things, up to the larger things, we get right into the file system IOPath, CAM, geom, buff, bio. We have a bunch of hacks for running a 32-bit userland on a 64-bit kernel. We don't actually need them anymore, but they're still there. We have
02:41
large changes to the kernel NFS server. We have changes to credentials. We have changes to venodops. We have changes to all sorts of things. And then, as well as that, we have our on-disk file system code, cluster backend networking, our internal userland stuff. These aren't as tied in to the rest of
03:06
it. They're not so much diffs against FreeBSD, but they rely on the changes that we've made in other places. So, the short version is we have a fork. So, why
03:21
do you fork? Well, in our case, it's fairly obvious. We have a whole bunch of constraints and use cases that don't apply to most uses of FreeBSD. I mean, the fact that our file system is actually distributed across multiple machines in a cluster is something that doesn't normally happen with FreeBSD. With other projects, I would actually, if you can manage it, I
03:44
wouldn't fork. The reason being that you're bringing on a whole lot of work, and that's part of what I'm going to be talking about, is just the amount of work that's involved in keeping this thing up to date. It also makes it tricky to push work upstream. It can lead to you not contributing stuff
04:02
upstream that you probably should, because you develop it internally, and you tailor it so much to your internal stuff that it's hard to get up there. So, while we have a lot of rationale for doing that kind of stuff, that if I
04:20
was starting up a new project, I would be working as hard as I can to try and make sure that I could self-contain my changes into things like kernel modules, or stuff that didn't involve actually forking. So we have a fork, and so as anyone who's maintained internal forks of
04:48
software can attest, there's always fun incorporating changes from the upstream project. If all you've got is a few minor tweaks, it's not so hard. If all you're doing is changing tuning parameters, not so much of a problem, but we've got much more extensive changes, and so hilarity tends to ensue.
05:06
Our previous pattern for doing these version updates, so Isilon's 1FS started out being based on FreeBSD 5. It got upgraded to FreeBSD 6, and so by that I mean it was a fork off the stable 5 branch. It then got
05:23
sort of uplifted to being a fork off the stable 6 branch, and then up to the stable 7 branch, and it never really made it past that. It's been sitting on being FreeBSD 7 based, but by FreeBSD 7 based I mean FreeBSD 7 plus an array of backports for quite a long time, and so this kind of gets me to the
05:46
first lesson that I kind of want to drag out of this this story that I'm telling here. Don't do that. Don't get behind. It can be really tempting to stay
06:00
where you are. Once you've sort of brought yourself up to a certain point and you've chased all the bugs out, you've got more project and feature projects you want to put resources on, you know, it can be kind of nice to be there. I mean one of our reasons was validation, you know, we sell a high value product to big customers. Well, you know, that makes us an enterprise
06:23
product. It comes with a heightened level of expectation that your product is actually going to be solid, and that means that you can have a bit of resistance to change, especially change to a large complex body of code that's not 100% yours. You don't have full control over it, and it's an integral
06:45
part of our system, but it's not fully under our control. One of the ways to increase that was just to not change it. So, and you know, that can lead to a certain level of comfort. The problem being that, you know, when you're in that
07:00
nice little comfortable place, you're suddenly several major versions behind. You know, we're on stable seven. You've hit the limits of what you can do with that code base, which we have. We're starting to run into problems that can really only be solved by moving up to later versions, and you've also done things like pull in and adapt revisions that fix immediate needs without any thought to how you're going to carry that forward, and you haven't
07:22
marked where your changes are, and you've gotten slack with marking your local modifications, and so it's really hard to differentiate between what's yours and what's something that's from upstream, and at that point, you're going to have a bad time. The problem in this way of handling
07:42
upgrades is it's kind of this big step approach. You're wearing a large amount of pain on a certain schedule. You start at stable five, and then you have to abstract out what your change is and then move it up to stable six and then do that again to get up to stable seven, and because of the delays and not doing it, we were looking at having to do that from
08:03
stable seven all the way up to stable ten, and that's not pleasant, and so avoiding the change just leaves you with a pile of tech debt. It just grows and grows and grows. So thus was born the merge project. So I joined Isilon's
08:25
platforms team in February last year, 2013. Platforms in Isilon is the main group that cares about FreeBSD within the product and the various bits related to that, so we also look after ports and sort of all the other stuff around that, and so it fell to the platforms team to do something about the
08:43
fact that we're on an outdated version of FreeBSD. When I joined, we were partway through the first iteration of that project. The process in this one, it was sort of mildly ambitious. We were going to take the
09:02
FreeBSD current. We were going to stick our file system and cluster backend and all the other sort of related code next to it, and then we were going to make it build. So then that would basically be you take the stock FreeBSD in our code and you bring in changes or adapt in changes
09:24
from our source tree as needed to make it work, and this sounds really great because it means we're clearing out a pile of dead code because we're only porting over the bits that we need. We don't necessarily bring over all the dead stuff, like say our 32-bit user land stuff. The problem is that
09:43
the questions that are being raised around it were how do you know it will work, and more importantly to a bunch of people, when will it work, and the problem is that the answers to both of those are really fuzzy. The when question is problematic because while you might be able to estimate the
10:01
resolution time of each particular issue you come across, you can't know how many there are until you don't hit anymore, and when you don't hit anymore does that mean there aren't any? And that then leads into the how
10:21
question, and that kind of underscores the darker side of this sort of removal of unused and unneeded changes is you can't be sure whether a change is actually unused or unneeded until you don't hit a problem with it.
10:42
It really comes down to this, and that's when it really comes down to the scope and the coverage of your test suite, and while in Iceland we've got a pretty good test suite, but even we'd admit that it could be even bigger and take even longer. Some of our tests can run for hours if not days, so the
11:01
problem is that our testing of FreeBSD itself has been more organic than systematic. We've generally gone, okay, we're hitting a bug in this, so write a test, okay, the test is showing that the bug is there, and now we fix it and the bug and the test now passes. Excellent. Or we're going to make a change here, so let's make a test to make sure that the change works. That's great, but it
11:23
means that part of the nervousness within the organization around this merge project was how do we know that we're not going to break something subtle in FreeBSD and then have a really bad time. When your entire business revolves around making sure that people's data stays put and
11:43
doesn't go away, you really want to be sure about this kind of stuff, and so the amount of change that this project embodied was just bringing up all that kind of nervousness and paranoia, and so we really needed to have a think about how we addressed that. So at one point, one of the
12:03
engineers had actually been involved in previous update projects in the company, spoke up and put up an alternate plan. The new plan involved instead taking our code tree, so the one that we had already, and then
12:23
iterating through FreeBSD changes, adding them until we got to where we needed to, using our internal test suites as validation. So this brings us to this one. The thing that this plan didn't do was kill off all
12:43
those unused and unneeded changes. On the other hand, we knew that we had all the changes there so that we were more comfortable with that we weren't losing anything, but part of our problem was we didn't keep our local changes under control. We've got really widespread changes, probably more than we actually need, and we've also been really bad at getting rid of the ones
13:03
that we actually do not need anymore. So for example, someone adds a bunch of diagnostic stuff to find a bug. Part of what we end up having to bring across is that diagnostic code that's still there even though the bug's been fixed. So I know that Netflix, for example, are a lot more aggressive at
13:24
keeping their diffs small than we are. They've got different usage parameters than we do, and that makes it easier for them. They also started doing this a bit later than we did, but still. So one of the things that this doesn't do is cut those changes out. The general point of view around this
13:46
version was that we just bring everything up to head, or at least somewhere that we were comfortable with, and then go through the resulting diff and start actively looking for things that we could upstream, delete, or deal with in other ways. And the other thing is that you've probably noticed that I've
14:05
been talking about moving it up to head. This is a great idea. It was part of the original plan as well. We went for a checkout of current rather than a checkout of stable 10 or stable 9 at the time, and it came out of discussions about how to avoid getting behind again. FreeBSD current over the years has
14:25
actually become alarmingly stable in a lot of cases, and various things have contributed to that. Some of them are good. I think over the years we've gotten better at the pre-commit testing of things before we actually put them in the tree, and some are less good at that. Less
14:41
things are happening, or less big things are happening. I mean, if you think back to when FreeBSD was at its most unstable in current, most people would probably think about FreeBSD 5. That was when we were trying to go through and completely fine-grain the locking systems to allow us to do proper SMP, and we haven't really had a body of work as disruptive and wide-ranging
15:03
in that in quite a while. Since then, it's actually gotten a lot more stable, so we figured if we combine that stability with our internal testing and release engineering, we reckon we could release off head, and thus be able to just track head on an ongoing basis, and I think this is just huge for us. It's a
15:27
great idea, and it's not just a great idea for us. It's a great thing for FreeBSD, because now FreeBSD will be getting like belted through our internal test systems, and I think I actually got a message today from one
15:42
of our guys that did the merge work that we'd actually found a years after freebug in the kernel in head through the testing that we're doing on our merge project. So, I mean, this is sort of like FreeBSD benefited for years for being the frontline server OS at Yahoo. I mean, that was a long time ago, but we got a lot of stability in that kind of serving class from being that,
16:04
and what I'm hoping is that as Isilon moves ahead with this project, we'll be able to contribute a similar sort of testing and proving environment for FreeBSD. So, back to the story. We decided to go ahead with Plan B. The
16:24
first thing we had to do was to work out a way to bring in the FreeBSD changes to our source tree. So, you know, obviously Git would be a perfect tool for this. Git allows you to merge from other branches, you know, you can rebase, you know, that kind of stuff, but naturally, like all modern
16:43
forward-thinking organizations, we use Subversion. So, Subversion, it's a thing. It can merge branches within a repository, and it can track the mergers that it's done, and so that's cool. It can't do that across
17:03
repositories. I don't know why the merge info format looks like it should be able to do that, but it doesn't. So, we wrote a script. We stole Subversion's own merge info format, and we just stuck a different version of it in
17:20
another property and used that to track revisions that we pulled in from FreeBSD, and then we did some archaeology to work out what FreeBSD revision we could reliably say we'd merged up to, and then we bootstrapped the merge info with that, and then we wrote a Python script using PySVN that had pull in revisions from FreeBSD head and
17:41
apply them into our repository until it hit a conflict, and we also did some other tooling as well. I've got previous history in writing web applications, and so we wrote a thing for tracking it, which management loved. So, the green ones are ones that we merged. The white ones we hadn't yet. The marked ones have skipped, and the bit that
18:03
management loved the most was over on the left there, where it showed them how fast we were working, because this was part of the win out of this was just being able to get a better idea on how long it would take once we'd actually done some of the work. So, we
18:22
started work and immediately ran into... So, like I said, we were on stable seven, and we were on stable seven plus a whole bunch of mismatched and adapted backports, and we're trying to get onto head. Now, git has a term for this. It's called a rebase. This was the ugliest rebase you ever had ever
18:45
thought of. We had our branch off stable seven. You try and pull in a revision from head, and half the time you would either have a variant patch. So, a change had gone into head and then had been adapted to merge back into stable seven,
19:04
and so then we had to untangle that. Or the most fun was because we would have pulled in a backport, we would be pulling in a change that moved a particular piece of code from version A to version B, but we already had version C, and so we would have to look at that and go, oh, so hang
19:23
on, we actually need to ignore that change because this is going to change further down the line. And in some cases we had version F, and so you'd keep hitting this conflict on this bit of code over and over again. That's lots of fun. And that slowed things down quite a bit in
19:43
the early phases of this. However, one of the things that we could do with this is run it through our existing test suite. This was great because we could prove that we weren't breaking things. And after a while using the tools that I showed you before, we could
20:02
establish a rate at which we were applying revisions and extrapolate an end date from there. There are only two problems. One is that there were changes to the way layer two addresses and routing were handled, and they totally blew up our cluster backend. We have an
20:26
InfiniBand stack that predates the one that went into the FreeBSD tree, and it made a whole bunch of assumptions as to how things worked, and then the L2 changes came along and it blew up. And that stopped
20:42
us from using the test automation because the test automation relies on being able to construct clusters, and when you don't have a backend network, you can't build a cluster. And instead of stopping and fixing everything there, we kept going. So we were still pulling in changes, and that meant that we had a
21:00
whole pile of changes that weren't tested. And then after the couple of months that it took to fix the InfiniBand stack, we then had to go through and fix everything. That wasn't the main thing that sank it. The main thing that sank it was it wasn't very fast. The merge process, as you can kind of
21:21
work out, was very single-threaded. Only one person could be doing it at a time because you needed to make sure that people weren't trying to pull in the same revision. We got around that slightly due to the fact that I live in Australia, and so I would take one shift, and the other engineer who was working on this was based in California, and he would take another shift. And
21:42
because our time designs didn't really overlap all that much, we could kind of be working for, say, 12 to 16 hours out of a day rather than just eight. But it still just, even after we got out of the stable seven era of having to deal with this version A to version B to version C stuff, we still weren't merging fast enough to
22:02
really hit where we needed to go. So we had to go back to the drawing board again. This second approach was not a total loss, though. We'd made enough progress that we'd mostly levelled out a lot of the mess of backported revisions
22:21
that we'd had, enough that we could do a diff between the revision we'd merged up to and the stock FreeBSD tree at that revision. So the last approach was take the diff we generated between FreeBSD and our code base and split it out
22:41
in roughly biological pieces. And then we took a check out of our source tree, stripped out every FreeBSD sourced file in it, replaced it with the version we wanted to get to, and then started applying these changes to FreeBSD so that we had a baseline that was a modified version
23:03
of FreeBSD. And then we put our code in next to it, and then we make it all work. So again, I wrote another web app, which I can't show you this one because it's got internal code in it. But that was more about tracking who was owning a particular
23:23
bit of the diff, whether it was merged, whether it had that kind of thing. So this might sound quite familiar to the first iteration that we went through of the take a clean FreeBSD, check out, and just make it work. The difference is that instead of just pulling in changes as needed,
23:40
we're pulling all of the changes in up front and then making everything work, which gets us around the nervousness around missing changes that we don't need. Once we're done, then we'll take what we were going to do and sort of rigorously audit the changes we've got and start trying to strip things out. And we're
24:01
also getting way, way more serious about testing things, and particularly around testing the FreeBSD elements of our system specifically. So one of our employees is Peter Holm. Peter Holm wrote the Stress II test suite,
24:20
and he's doing that for us. He's contributing a whole bunch of stuff back to FreeBSD as well. This is going to be a theme, contributing back to FreeBSD. So we're using that on our stuff. He's also written some internal stress tests for us. We're getting really serious about ATF and CURE.
24:43
We're running, we pulled in the ATF and CURE stuff. We also pulled in a lot of the work that Garrett Cooper's been doing importing tests over from netBSD. We're also getting people to write us new tests, and we're hoping to be able to contribute a whole lot of those back up to FreeBSD, which is awesome.
25:04
And we're also ensuring that as much of our internal validation, we can run against every build that we've got, even if that means we have to stub things out early on. Once they start to pass, then we can leave them turned on, and it's great. And it's paying off. We're finding bugs a lot faster. Like I said, we found one in HEAD earlier, which is great for us
25:23
because we know that we're not busting things, and it's also great in placating the same sort of people who get nervous about the fact that we're changing all this. So that leads to another lesson for me, which is back when things stopped and
25:41
sort of decisions were sort of, I'm guessing I wasn't around for it, but it is my feeling that it's not so much a decision that gets made, it's more of just a step that's not taken. When you fall behind in this kind of thing, it's usually that you go, oh, we don't need to move up.
26:00
And you don't actually stop to think about why it is you're not doing that. If validation is the reason that you don't want to upgrade, if you're worried about it, then fix the validation. Don't just decide that it's safe and comfortable where you are, because then you're just racking up the tech debt. So solve the problems that actually stop you keeping current, as opposed
26:20
to just deciding that it's okay to not do it. That's more of a 2020 hindsight thing than something that's obvious at the time, but that's how it goes. So the project is still in train. I can't tell you that it's been a smashing success. It's going well, but it hasn't finished yet.
26:41
So I'm very much looking forward to us shipping it. I think we're going to make it. I think we're doing good. But what happens after that? This is the other reason why it's a good idea to track head and not a stable branch, because head's always going to be alive. The problem if you're tracking a stable branch
27:00
is if you do stay on that stable branch, eventually that stable branch will stop getting actively developed. It'll get mothballed, and then you're not getting updates. With the tools that we developed for our revision-based approach, the Plan B, and the tests that we're adding in, we're planning to be able to pull in a whole bunch of head at the start
27:22
of each release cycle internally, put it into our head branch, and then that gets then tested and proved out during the development cycle, which means that we can actually stay a lot closer to the top of current than we do now. And that also means that we don't have to do these huge uplift projects from stable X to stable Y.
27:43
And it also means that we're in a much better position to do things in FreeBSD directly, because suddenly the payoff for having developed something for FreeBSD is a lot faster. Historically, we'd have to wait until it hit the stable branch that we wanted to use, and so the timeline before contributing a feature
28:03
to FreeBSD and us being able to use it was just too long. Now, if we're tracking head, we can do things like add features to FreeBSD in various ways, and then pull them back into our product and use them there.
28:20
So the final lesson is really support the community. If you're gonna put a community-based project at the core of your product, supporting that community, whether you like it or not, is now a really high priority for you, because if you don't support it, you run the risk of the community going away, and suddenly you own in the sense of being 100% responsible for a lot more code
28:41
than you used to. In our case, FreeBSD is a big thing for our product, and we're working on getting a lot better at making sure that we give back. So this will be taking the form of developing things in FreeBSD directly, doing a lot more testing. I'm also looking into how we can contribute to making FreeBSD itself more tested.
29:00
The QA and ATF stuff that Julio Moreno is doing is great. The Jenkins stuff that Craig Rodriguez is doing is great. What I'd love to see and what I'm hoping to get them to work towards is having Jenkins building a virtual machine image that then gets booted and QA gets run on it. If we could have QA being run on a live instance
29:22
of FreeBSD several times a day, that's brilliant. And yeah, that's why we're contributing to the testing stuff, and if we can contribute more tests than that, then that's just getting FreeBSD more tested. It makes FreeBSD current more stable for us,
29:42
and it makes it a lot easier to just keep pulling changes in from FreeBSD because we know that it's gonna be there for us. So basically, in conclusion, don't get behind. You'll really regret it. We are regretting it.
30:00
If you are tempted to drop behind, try and work out why that is. Keep your local changes under control. Minimize your diff as much as you can. That can mean upstreaming, that can mean deleting things, that can mean adapting things to try and use other parts that require less change on your part. Where you do change things, mark them.
30:22
Part of what we're doing is going through and just making sure that all our changes are marked. You can use comments, you can use ifdefs, all that kind of stuff, but just make sure you know what your changes are. Having documentation on what your changes are and the intent behind them would be great, but let's be realistic here. If you can, track head.
30:41
Some people don't feel comfortable doing that. I would recommend it. I think it's the best way forward because you're in a much better position to contribute. You don't have to have these uplift projects, so on and so forth. Solve the problems that stop you keeping current. Like I said in the first one,
31:00
if you feel tempted to not upgrade, try and work out why that is and make sure that you're giving back because otherwise the community might not be there anymore. And that, while short, is what I have. Thank you.
31:23
Questions?
31:52
Well, sorry, the question was, given that we're an enterprise business,
32:02
how are we going to make sure that our customers are applying updates? So our current plan for a release cycle is at the start of the release cycle, we'll drop in as much change from head as we feel comfortable to get as much of the new stuff in. Our release cycles tend to be in the order of,
32:25
well, over a year. And when the upgrades go out, they'll have that change in them. Obviously, if we hit bugs that we need to pull in revisions from head to fix, they'll get rolled into those bugs and sent out.
32:40
But the release cycle of 1FS is kind of detached significantly from the release cycle of 3DSD. Yeah, and because we need to be able to stabilize and prove out what we have.
33:01
And so that means at some point we do have to stop taking changes. So release X will have changes up to revision A. Release Y will have changes up to revision B. And so on and so forth. And there may be some cherry picked revisions from further ahead of that, that we just need to fix bugs and whatnot.
33:21
Any other questions?
33:45
So the question was, how do we decide what head revision we're targeting? With the one we've got now, the thought process was, we want something close to the 10 release. So we don't want the stable 10 branch point.
34:03
We want post the stable 10 branch point. So we want the fixes that had to go in to get the 10 release stable. But we also wanted a couple of features that came in there, which was mainly the revised locking around cam and the direct dispatch stuff for geom. And so the revision was picked based on that.
34:26
Future releases will probably be sitting down and hanging right. So if we work our way back from where head is right now, is that a safe place to do it? Or should we take it further back? Or are we waiting for something to come in or so on and so forth?
34:41
I think it's gonna be a discussion each time, but it's gonna be a discussion that we have sort of once a year or thereabouts. Any other questions? Yes. Yes.
35:00
Yes. So the question was, so current is going to be less stable than stable. So how do we convince management? Thankfully, a lot of our management were quite happy to go with the plan,
35:23
especially when, because one of the things was, how did we get so far behind and how do we stop it happening again? And so one of the answers to how do we stop it happening again was, well, let's stop sticking to stable branches. Because then you just have this,
35:40
like I said, these big uplift projects to get from stable X to stable Y. And so if we can stop having to do that, the question then becomes, how do we then validate what we've got? And that's where all this extra testing infrastructure and stuff is coming in and just making sure that we can actually prove that what we've got is stable and does work. And so that means just test all the things basically.
36:07
Anyone? Thank you very much.