How to replace a legacy tool with 100k installations
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Untertitel |
| |
| Serientitel | ||
| Anzahl der Teile | 84 | |
| Autor | ||
| Lizenz | CC-Namensnennung 2.0 Belgien: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen. | |
| Identifikatoren | 10.5446/40024 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache | ||
| Produktionsjahr | 2012 |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
|
FOSDEM 201245 / 84
1
2
3
5
7
9
11
13
14
15
16
17
18
20
23
24
26
27
28
30
32
37
39
41
44
46
47
50
51
52
53
55
57
58
63
68
70
73
75
79
80
81
84
00:00
SoftwareentwicklerSchreib-Lese-KopfInstallation <Informatik>ZahlenbereichReverse EngineeringSoftwarewartungSkriptspracheBenutzerfreundlichkeitInstallation <Informatik>ZweiXMLUMLVorlesung/Konferenz
00:47
SoftwareentwicklerSchreib-Lese-KopfInstallation <Informatik>GenerizitätBeobachtungsstudieSoftwarewartungSpieltheorieSkriptspracheDistributionenraumCASE <Informatik>BeobachtungsstudieProzess <Informatik>MAPJSONXMLUML
01:46
BeobachtungsstudieGenerizitätMessage-PassingProzess <Informatik>Message-PassingTOEVorlesung/KonferenzJSONXMLUML
02:17
Message-PassingSoftwaretestYouTubeZahlenbereichDeskriptive StatistikBasis <Mathematik>Installation <Informatik>SoftwareentwicklerWellenlehreXMLUML
03:14
BeobachtungsstudieRestklasseServerAbstimmung <Frequenz>Installation <Informatik>SkriptspracheMereologieUmwandlungsenthalpieDienst <Informatik>Elektronische PublikationImplementierungATMKonfigurationsraumCybersexKlasse <Mathematik>Chirurgie <Mathematik>MultiplikationsoperatorServerBimodulProgrammfehlerStandardabweichungSoftwarewartungSkriptspracheFormale SemantikVorlesung/KonferenzXMLUML
05:13
SteuerwerkMathematikMeta-TagRechenwerkQuaderSoftwareentwicklerMIDI <Musikelektronik>Streaming <Kommunikationstechnik>ComputerspielProgrammfehlerVorlesung/KonferenzXMLUML
06:22
SteuerwerkCodeSkriptspracheServerKonfigurationsraumVererbungshierarchieVorlesung/KonferenzProgramm/QuellcodeJSONXMLUML
07:57
Rechter WinkelMultiplikationsoperatorVorlesung/Konferenz
08:30
AggregatzustandKette <Mathematik>DistributionenraumFakultät <Mathematik>MathematikTabelleProgrammierungJSONXMLUML
09:03
Güte der AnpassungMailing-ListeMultiplikationsoperatorE-MailWellenlehreMereologieVorlesung/Konferenz
09:39
Vorzeichen <Mathematik>NormalvektorImplementierungBeweistheorieCodeIterationDeskriptive StatistikMomentenproblemÄhnlichkeitsgeometrieNichtlineares ZuordnungsproblemSoftwaretestPlastikkarteVideokonferenzJSONXMLUMLProgramm/Quellcode
10:47
IterationBeweistheorieGruppenoperationSoftwarewartungCASE <Informatik>ImplementierungValiditätBeweistheorieVorlesung/Konferenz
11:35
Vorzeichen <Mathematik>NormalvektorImplementierungBeweistheorieIterationProzess <Informatik>MultiplikationsoperatorCoprozessorVorzeichen <Mathematik>ImplementierungJSONXMLUML
12:29
E-MailVorlesung/Konferenz
13:00
Wurm <Informatik>Prozess <Informatik>Weg <Topologie>Strom <Mathematik>ImplementierungBeweistheorieProzess <Informatik>EinsMultiplikationsoperatorDistributionenraumCASE <Informatik>MereologieEinfache GenauigkeitImplementierungPackprogrammÄhnlichkeitsgeometrieWeg <Topologie>Mailing-ListeArithmetische FolgeUMLFlussdiagramm
14:59
PunktMultiplikationsoperatorÄhnlichkeitsgeometrieEinfache GenauigkeitWeg <Topologie>ImplementierungVorlesung/Konferenz
16:52
Prozess <Informatik>Weg <Topologie>GruppoidServerAbschattungNichtunterscheidbarkeitQuick-SortNichtlinearer OperatorUmwandlungsenthalpieKonditionszahlProgrammierungUnendlichkeitUMLFlussdiagramm
17:27
Kontextbezogenes SystemServerAbschattungMatchingKontextbezogenes SystemErwartungswertElektronische PublikationProgrammierungUnendlichkeitKonfigurationsraumVerzeichnisdienstSoftwaretestComputeranimation
18:24
GruppoidGruppenoperationMeta-TagKontextbezogenes SystemErwartungswertAbschattungServerRechenschieberTermFolge <Mathematik>EreignishorizontEin-AusgabeUmwandlungsenthalpiePhysikalisches SystemMultiplikationsoperatorFolge <Mathematik>TermCodeCASE <Informatik>EreignishorizontSchnittmengeRechenschieberProgrammierungSoftwaretestFunktionalBasis <Mathematik>ImplementierungKontextbezogenes SystemKonditionszahlProgramm/QuellcodeXML
19:41
Kontextbezogenes SystemEreignishorizontTermFolge <Mathematik>RechenschieberMereologieSoftwaretestFunktionalInstantiierungProgrammierungCodierungCodeDeskriptive StatistikFolge <Mathematik>PunktNormalvektorKontextbezogenes SystemImplementierungStrategisches SpielLokales MinimumVorlesung/Konferenz
21:19
RechenschieberTermFolge <Mathematik>Kontextbezogenes SystemSoftwaretestPhysikalisches SystemGruppoidGruppenoperationErwartungswertServerAbschattungMeta-TagSoftwaretestCASE <Informatik>TabelleElektronische PublikationXMLUMLProgramm/QuellcodeVorlesung/Konferenz
21:57
RechenschieberTermFolge <Mathematik>Kontextbezogenes SystemEreignishorizontPhysikalisches SystemSoftwaretestRechenwerkSummengleichungKanal <Bildverarbeitung>Elektronische PublikationGruppenoperationProgrammierungSchaltnetzXMLUMLProgramm/QuellcodeVorlesung/Konferenz
22:45
ServerAbschattungGruppoidGruppenoperationKontextbezogenes SystemErwartungswertMeta-TagParametersystemProgrammierungKontextbezogenes SystemProgramm/QuellcodeXML
23:11
AbschattungServerCodeBildgebendes VerfahrenElektronische PublikationProgrammierungKontextbezogenes SystemServerMatchingDeskriptive StatistikCodeSchaltnetzFunktionalSkriptspracheParametersystemTouchscreenLoopSondierungSoftwaretestDienst <Informatik>XML
24:33
SoftwaretestSoftwaretestArithmetisches MittelImplementierungForcingArithmetische FolgeFunktion <Mathematik>Güte der AnpassungProgramm/Quellcode
25:03
SoftwaretestArithmetische FolgeFunktionalMultiplikationsoperatorKartesische KoordinatenImplementierungVollständigkeitMAPProgrammierungSchreib-Lese-KopfComputeranimationVorlesung/Konferenz
26:01
Zeiger <Informatik>Zeiger <Informatik>Quick-SortSoftwaretestPhysikalisches SystemKontextbezogenes SystemDateiverwaltungProgrammierungKomponententestCASE <Informatik>MathematikElektronische PublikationSystem FErwartungswertJSONXMLUMLVorlesung/Konferenz
26:42
Zeiger <Informatik>MenütechnikVerschlingungSoftwaretestProgrammierungFramework <Informatik>Physikalisches SystemImplementierungUbiquitous ComputingJSONXMLUML
27:13
DatenmodellRechenwerkDokumentenserverCodeDistributionenraumSoftwareentwicklerExogene VariableVorlesung/KonferenzXMLUMLProgramm/Quellcode
28:35
E-MailRückkopplungVorzeichen <Mathematik>ComputerspielMathematikGüte der AnpassungMaschinenschreibenImplementierungVisualisierungMultiplikationsoperatorSoftwareentwicklerPunktVerkehrsinformationTeilmengeMAPProgrammfehlerObjekt <Kategorie>EnergiedichteSoftwarewartungAdressraumVorlesung/Konferenz
Transkript: English(automatisch erzeugt)
00:01
Hi everyone, I'm Serfi, I work for Deviantart. I'll be talking about how to replace the legacy maintainer tool with a number of reverse dependencies. Just to show of hands, how many people here know what a maintainer script is?
00:22
Ok, so I'll just spend a few seconds explaining. When you install a package, or when you uninstall a package some packages can include the so called maintainer script. So when you install the package, the package installer
00:42
is actually running the maintainer script. So I'm talking about a tool which is used by about 50 maintainer scripts. And it's a legacy tool, so it's something that we'd like to replace.
01:01
And the challenge is getting to coordinate with the 50 package maintainers to do that. So the audience at the top is basically maintenance of community distributions. And by that I mean distributions where there is no
01:22
benevolent data that says we will do that. It's about distributions where you have to convince people you have to get consensus to do something. And the tool I'm talking about, the case study is very specific, but I'm drawing some lessons which are
01:40
I think of general interest. So this is not to talk about describing the replacement. It's to talk about the process of getting consensus for replacing the tool. And I've planned to make it a show talk, so I encourage you to interrupt me anytime with questions.
02:02
Otherwise I'm going to have everyone else. And this is the message I'd like all of you to go away with at the end of the talk. So the idea is that in replacing the tool
02:22
with the last number of installations and the last number of request dependencies, something that checks a lot is having a design document to describe the new tool. And having a written description of the new tool is key for getting to consensus,
02:43
so for convincing the people that we have to update the packages to use the new tool. And since we also have a design document, that's also a very good basis for writing the tests for the new tool.
03:02
I think this is something that Joy has, one of the very influential development developer. His tools tend to be very well documented, and I think that a big reason for that is that he actually writes the documentation
03:21
as part of doing the design of tools. So this is a specific example I'm talking about. Updating a D, it's a per script and a module, which was written a long time ago.
03:41
As I said, it's invoked by maintainer scripts that have server packages that can be run via in a D. I don't know whether, how many people here know in a D? Okay, more. So it's basically a server you can consider
04:00
you can work out a server. So when you install, for instance, an SDP packet, it will have a script that will be run by D packets to call update in a D, and update in a D edits the configuration file in a Dconf, and restarts in a D. So this is a very old tool.
04:22
It's becoming increasingly relevant because most people nowadays can run servers in standard or more, so in a D itself it's not that bad or not. But it's still very widely installed and used, and it's quite valuable. So in my opinion, and not only why,
04:42
it's kind of broken by design because to edit the configuration file, you can only specify the service name, and that is not sufficient to uniquely identify entries because you can have more than one implementation on the same service, so you can have IP, V4, and IPv6 services.
05:01
So there is a certain class of bugs that really cannot be fixed in this implementation without really changing the semantics of the way you invoke it. So a few years after it was introduced in Debian,
05:20
somebody opened a bug, so that's a very old bug, saying you can only add one entry. And why is this interesting? Because these bugs are still working, and this package is still in Debian. And a few years later,
05:40
the prominent Debian developer sent a proposal for re-identity, and then the lively discussion followed, which didn't seem to lead to any problems.
08:17
A year later, I got kind of interested again,
08:21
so I got another proposal, but this time, the right way. So the question is, what is the wrong way of making a proposal about introducing a chain which impacts many other package users?
08:44
And this is the difficult question in community distribution, because there's a significant state where the data will say, we do this, and then everyone does it. But in community distribution, somebody needs to come up with a proposal and convince everyone.
09:01
And this is one of the big problems in Debian, that anyone should accept. So the worst way of making a proposal is to just send an email to the largest mailing list with a half-baked idea. So something is going to find out, something is going to accept,
09:20
something is going to send it to the largest mailing list, and then sometimes it's not going to be good because you haven't been told about it, but the people that are directly impacting you will also be upset because they're going to deny you even if it's not going to do that. So this is how I've done it, more or less the first time. Not as bad as,
09:41
I've discussed it in a few videos, but it's not enough. And so what is the right way of doing it? Well, this is why I came to talk it up. So, lastly, design documents. Thinking about a problem in Apps5.
10:01
It's very easy to think that you keep it out of the detail, but the moment you're actually forced to put down a little description, you realize all the issues that were not apparent when you were thinking about it, as opposed to writing a detailed description.
10:25
So once you've designed documents, you can bounce the idea of someone that's close to you and that thinks back to you after having seen your proposal or something similar.
10:40
So once you've passed this basic scientific test, then you can try to validate the main assumptions of the implementation. So do a group of concepts of the implementation, not a whole daily implementation, just to verify the basic assumptions of the implementation.
11:00
And to both allow you to do a group implementation, so you do a group implementation, then you will be invested in this idea. You will be too attached to be open to criticism when you do this idea. So once we know that it worked and we thought that it did this idea,
11:20
then we can actually engage into the discussion of the people that have been directly impacted by this claim. In the case of updating, it did have to be a maintenance of the ID package and the actual ID package. And this is an iterative process. In every step, you need to refine it and adapt it.
11:47
And so if you stay fine with it, then you can actually send it to the lab that's naming it and really track the process efficiently. And as I said, this is very much an idea.
12:04
So, some honest remarks. If you were getting it, this is a good sign. The last one is not so obvious, but it's very tough. Being ignored until you show some goals is normal. It's tough because preparing the idea
12:25
and taking it away and bringing it to the concept of implementation only takes time. When you're thinking of doing all this work, people need to be appreciative of it. But the thing is that it's so easy to just write an email with ideas
12:43
and then not follow up. It happens so often that it's normal to be drawn. So until you actually show some goals about the proposals, it's possible that people won't be able to follow them.
13:01
And so because this process can take a long time, as I said, the 2001 proposal to replace update with a new, I couldn't find out what happened in that time. So mainly the times alone are not good enough
13:21
to be able to keep track of the progress of this kind of work. So in Daniel recently, some white people have introduced the Python and Haskell proposal. Which is basically very simple. It's just you do your, the discussion you're doing normally,
13:43
whatever, mainly. So it's actually the, it's not a dead situation, it's a document, which you can't say. It is a driver. So it's the person that drives the proposal, not necessarily the implementation.
14:01
And it is a status of whether it's still a draft, whether there's been consensus, whether the implementation has been tested, then it's actually part of the distribution. So this is useful to have so that if something has been tried before and failed,
14:21
you don't have to look into the archives. There is a single place in Debian. All the Debian proposals are here. So you can go there and see the things that have been tried before and worked or didn't work, and why. And I don't know whether there are people from other distributions here
14:41
to say whether there's something similar in their case. I know this is more the laughter Python, the Python enhancement process. Okay, no people from other distributions?
15:21
And is there a single place? Yeah, I guess I have to repeat because you're not on the mic. So he's saying that there is something similar in OpenSUSE for keeping track of proposals and the status. Is there a single place where, so is that Mozilla or SUSE?
15:58
So there is a disagreement here.
16:03
So it's not working exactly. Yeah, okay, so this is, even in Debian, this is optional. You don't have to do it. But it doesn't take a lot of time and it's helpful.
16:35
Okay, so hopefully by this point we have a proposal.
16:43
We have consensus. So we can now be confident that we can proceed with the implementation. And if we have a depth or a design document of some sort, then we should have some sort of summary of operations. So a specification of what the tools should do under certain circumstances.
17:04
This is the summary of reconf in a D, which is the replacement of update in a D. I'm not going to describe the specifics of the tool because that's not the subject of the talk. All I want to say is that if you have a design document, it's helpful to have a summary of, given some conditions,
17:25
how should the program behave. So it's a very simple idea. Yeah, so that is the context in which the program operates, and that is the expectation.
17:41
So reconf in a D edits a configuration file. So depending on whether a file exists somewhere, whether the entry is there, and what's the status of the entry, whether there are some other files in other directories, it will remove an entry from the file, it will enable an entry, add an entry, or do nothing.
18:02
So it's that simple, but having it there, it's very useful for people to look at it and understand what it's about. And now I'm going to do a small parenthesis about behavior-driven testing.
18:21
So I said in the beginning that a design document is key in making, in getting to consensus, but once we get to consensus and we can start implementation, once we have the design document, we can actually use it as a basis, as an input for system tests.
18:43
So that's why I will talk about behavior-driven testing. It's only one slide. So behavior, how are we doing with time actually? Okay. Behavior-driven testing, it's a specific case of test-driven testing, so you write the test before you write the code.
19:03
And in behavior-driven testing, you describe program functionality in terms of features, and every specific use case of a feature is described in a scenario, and every scenario is described in terms of three steps, given when done.
19:21
So given a specific set of conditions, a context, when a specific event happens, then this is the expected outcome. And the way it works is that first you describe the functionality, you describe all the features in a sequence of scenario,
19:43
and every scenario is a sequence of steps. So you start by describing the functionality, and then for every step, you write the method that implements what that step describes.
20:01
So at this point, we have the human readable description, and the code to which that description is mapped into. We haven't written any code yet, any code that implements the actual program. So when we run this, the tests fail, which is normal. There's nothing there yet,
20:21
which is great because we have to see the tests failing first, because it's very easy to write tests that give a false positive, that succeed, even though because they test the wrong thing, for instance. So once we see the test failing, then we actually write the minimal code of implementation
20:40
to make that test succeed. And once we write the code that makes the test succeed, then we write the next step definition, we write the next part of functionality, the second part of the strategy. I guess I should say that
21:01
so the given is the part that sets up the context where the program, in which the program will be tested. So it's like setting up a fixture. And this is, of course, asserting that a certain outcome has occurred. Okay, so now that we are behavior-driven testing wizards,
21:23
we can go back to our use case. We are updating a D. So this is an example of one feature. Remember in the summary table, we have four features. It removes entries, it enables entries,
21:40
it adds entries, it doesn't do anything. So for each feature, we have a feature file and the remove feature has two scenarios, two scenarios, one scenario, we don't know how many. So for every feature, we have one feature file.
22:02
And in that feature file, we have one scenario. Well, we describe, we list all the scenario. So these are almost identical, but they both have the same outcome. They remove an entry. So we do this for all the features.
22:21
But it's kind of boring to do this by hand, especially for the no action. So these are just five scenario, we can write them by hand. But for the no action, it's boring and painful to write them by hand because you don't know all the combinations in which it shouldn't do anything.
22:41
So instead of writing it by hand, what we can do instead is actually write a program that will write the scenario for us. So what this program does is that it knows the valid values for every context parameter.
23:00
So if I go back here, so this is a context parameter, this is a context parameter, and these are the valid values. So the program knows the valid value for every context parameter, and it produces all combinations. And in the body of the loop, we know which combination
23:21
should add an entry, and so on. And this is useful because as we develop, we might find a new like a new parameter in the context, so we don't have to do everything by hand, we just update the script.
23:44
And so once we have the description of the features, then we have to write the code for the steps. So this is an example of two steps. This is a given step, a context step.
24:00
So it says given a matching server file that exists. It basically creates an empty file with that server path to make the program think that the server file exists. And the opposite, given a matching server file that does not exist, it's just an assertion that the file is not there.
24:21
So we write the code for all the steps, we run the tests, we see them fail, then we write the functionality, the actual functionality of the code, and we keep doing that until all the tests pass. So okay, this is the output of the tool when you run it.
24:40
When it's all green, it means it all passes, that's good. And the nice thing about this is that while it's still work in progress, you can see how far you are in implementation because you have a specific target. You have four features, four scenarios. So when you start, you have zero features passed,
25:03
and you do some work, it's like, so, you can quantify the progress of implementation, which is something that's, it's not really easy, it's like when you write the program, you have the functionality in your head, but it's, you have a very vague idea
25:21
of how far you are. But this way it's very specific, and it's also specific in the application level, not in the implementation sense. So I did the implementation, it took me one year because I didn't have time, I was only writing it during holidays
25:42
and vacation time. So this actually kept me, gave me a sense of progress. So it was a way to motivate and to have an idea of how far I am, of how close I'm getting to completion.
26:03
Just a couple of pointers. So these are tools for writing sort of behavior-driven system tests. I should say that these tests are system tests, so they're not unit tests, which means the test sets up
26:21
the context in which the program should run. In this case, the context is specific files in the file system. It runs the program, and after the program exits, it checks in the file system to see that it did the necessary changes. In this case, it edited in a D config, as expected.
26:43
So in my case, I'm using a Python framework, and the implementation itself is in Python, reconfinite D, but because they are system tests, it could be anything. It could be that the program is implemented in C or whatever. So the fact that these are in Python and Ubi doesn't really matter.
27:03
And finally, for those that actually do care about DEP9 and the replacement of updates in a D, a couple of links, the design documents and the actual code repository. Yeah, it's complete.
27:21
I just need to add a man page and hopefully upload it to experimental one of these days. So I would like to close by saying that Debian and other distributions are full of such legacy things that can be improved by quoting Zach,
27:42
beloved DPL, who said that the most fun way to contribute to Debian, just find something that needs improvement, take responsibility for it, and just do it. That's it. Thank you.
28:06
Any questions? Can we have a mic there?
28:22
Thank you. The last sentence, and do that, that's nice and all, but I'm trying to do some development on a Debian package, and doing that, that's one, but uploading and really contributing to Debian is two, and all the politics make it very hard.
28:44
So have you got tips for me to embrace the political Debian stuff and to make life more easy for me to upload changes? So to upload changes as, so you're not already maintained, right?
29:01
Yeah, to be honest, I'm not sure even. When you search my package, my name is there, my email address is there, but I don't know how to upload changes. Someone else always does that for me. Okay, so you're being sponsored by a developer?
29:22
Yeah, but I made some new changes, and my sponsor, he isn't answering my emails now, so now what? We have to find you a new sponsor, no? Can you? So Enrico says email Debian Mentors, but yeah, you can just grab me as well,
29:41
that's fine. We can talk about this after the session. Please, yes. Okay. Well, thanks. Thank you. Any other questions? I guess everything was clear. There's a question here.
30:10
How is the reception so far, Roberto? Did you manage to raise an interest? Do you have people following and asking you to finally upload it?
30:21
Good question. So, of depth nine, what happened is that before announcing it, I actually sent it privately to the key people that would be impacted by the change,
30:41
and also to the people that were very much against the first incarnation of the proposal. So I gave them the opportunity to give feedback or to object before I would announce it to the group. So all of them, basically, didn't get back to me, except the maintainer, Marco Dietri,
31:00
who was instrumental in refining it and improving it. So, and it actually, it wouldn't be at this stage, the design, if it weren't for him. So that was like a year ago. I haven't heard back from him since then, but in the meantime, I've announced a depth. Nobody was screaming loud against it, so that's my understanding
31:22
of, okay, nobody minds, so I guess it should be okay. And then from some point onwards, well, I've always been receiving bugs that the main motivation, bug reports for the update in the data, the main motivation for the new implementation, and I've always been saying
31:41
I'm working on the implementation, and from a certain point onwards, once depth nine was public, people were starting asking, when is it going to be ready? So that's an implicit sign of acceptance. Okay, Marco is here somewhere, so you may want to find him later. Yeah, I'm sure that he's here, but I don't know how he looks like, so.
32:02
I'll try to put you in touch if I see both. Okay, that's great, thanks. And it's probably time to be bold, because I went like, oh wow, you're doing that, that's cool, I didn't know about it, so yeah. Great, thanks. Okay, I guess we're done.
32:22
Give what?