FreeBSD Test Suite
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Serientitel | ||
| Anzahl der Teile | 24 | |
| Autor | ||
| Lizenz | CC-Namensnennung 3.0 Unported: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen. | |
| Identifikatoren | 10.5446/15363 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache | ||
| Produktionsjahr | 2014 | |
| Produktionsort | Ottawa, Canada |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
|
1
5
7
9
12
13
14
15
17
18
19
21
23
00:00
SichtenkonzeptLesezeichen <Internet>BildschirmfensterElektronische PublikationProgrammpaketStatistischer TestGleitendes MittelCOMSignifikanztestSuite <Programmpaket>HilfesystemGoogolStichprobeCodeElektronischer FingerabdruckProdukt <Mathematik>Produkt <Mathematik>Leistung <Physik>StandardabweichungRechter WinkelProjektive EbeneMultiplikationsoperatorÜberlagerung <Mathematik>MittelwertGrundraumCASE <Informatik>CodeEreignishorizontInternetworkingWarteschlangeStrukturierte ProgrammierungSchreiben <Datenverarbeitung>Fundamentalsatz der AlgebraWort <Informatik>SoftwareentwicklerSignifikanztestOrdnung <Mathematik>Quick-SortStatistischer TestAutomatische HandlungsplanungSchlussregelRechenzentrumSchreib-Lese-KopfStichprobenumfangOffene MengeSystemprogrammierungHilfesystemMereologieSuite <Programmpaket>ProgrammpaketInformationsspeicherungKernel <Informatik>Coxeter-GruppeProgrammierumgebungSichtenkonzeptWeb SiteProgrammierungNabel <Mathematik>TopologieXML
07:52
GoogolSuite <Programmpaket>SignifikanztestElektronische PublikationLesezeichen <Internet>SichtenkonzeptBildschirmfensterCOMEmulationHilfesystemCASE <Informatik>SignifikanztestBasis <Mathematik>WhiteboardMultiplikationsoperatorSichtenkonzeptVirtuelle MaschineZahlenbereichTopologieRechter WinkelEndliche ModelltheorieBeobachtungsstudieStatistischer TestCodeExogene VariableKondition <Mathematik>SoftwareentwicklerMobiles InternetFaktor <Algebra>VerzeichnisdienstProgrammbibliothekSystemprogrammierungServerVersionsverwaltungNatürliche ZahlElektronische PublikationFormation <Mathematik>Deskriptive StatistikKategorie <Mathematik>Wort <Informatik>Extreme programmingSchreiben <Datenverarbeitung>FrequenzDimensionsanalyseBitBereichsschätzungAggregatzustandFramework <Informatik>MinimumKontextbezogenes SystemPunktSuite <Programmpaket>QuaderSoftwareQuellcodeProjektive EbeneProgrammfehlerInhalt <Mathematik>Strukturierte ProgrammierungErwartungswertProgrammierungDifferenteNetzbetriebssystemHardwareProgrammpaketFlussdiagramm
15:44
COMProdukt <Mathematik>GoogolSuite <Programmpaket>SignifikanztestStatistischer TestInstallation <Informatik>Explosion <Stochastik>SichtenkonzeptElektronische PublikationLesezeichen <Internet>BildschirmfensterEmulationDatenstrukturSystemprogrammierungDifferenteMultiplikationsoperatorEntscheidungstheorieStatistischer TestWort <Informatik>SignifikanztestRechter WinkelQuick-SortElektronische PublikationVersionsverwaltungResultanteComputerspielMereologieSchreiben <Datenverarbeitung>Funktion <Mathematik>Lesen <Datenverarbeitung>VerzeichnisdienstBAYESWellenpaketTragbarer PersonalcomputerGenerator <Informatik>AbstandKorrelationsfunktionBildschirmmaskeBitrateFrequenzGebäude <Mathematik>CASE <Informatik>TopologieKontingenztafelDialektGeradeSpieltheorieViewerRechenwerkTypentheorieProgrammierungDeskriptive StatistikQuellcodeMailing-ListeMessage-PassingSuite <Programmpaket>VerkehrsinformationKontinuierliche IntegrationLaufzeitsystemInterpretiererInstallation <Informatik>ServerKernel <Informatik>BinärdatenAblaufverfolgungKomplex <Algebra>DatenbankZweiZentralisatorHierarchische StrukturStrömungsrichtungKonfiguration <Informatik>BitHyperbelverfahrensinc-FunktionMAPXMLComputeranimation
23:37
Statistischer TestGruppenoperationSignifikanztestNotepad-ComputerCAMElektronische PublikationSichtenkonzeptLesezeichen <Internet>BildschirmfensterGoogolCOMSuite <Programmpaket>DatenstrukturCLIKonfiguration <Informatik>Metropolitan area networkHilfesystemSummierbarkeitMailing-ListeVerkehrsinformationExplosion <Stochastik>FreewareElektronischer FingerabdruckURNGewicht <Ausgleichsrechnung>Quantisierung <Physik>RohdatenFehlerkorrekturmodellHochdruckCASE <Informatik>Kategorie <Mathematik>GamecontrollerSignifikanztestHalbleiterspeicherWarteschlangeDichte <Physik>OvalMereologieTopologiePhysikalischer EffektRechter WinkelWärmeausdehnungElektronisches ForumFunktion <Mathematik>PunktStatistischer TestProdukt <Mathematik>Ordnung <Mathematik>StichprobenumfangStrukturierte ProgrammierungNummernsystemVerzeichnisdienstFortsetzung <Mathematik>Arithmetisches MittelNeuroinformatikBitAutomatische HandlungsplanungProgrammbibliothekAutorisierungDoS-AttackeSchießverfahrenMomentenproblemKontrollstrukturVirtuelle MaschineResultanteSchreib-Lese-KopfWellenpaketSystemprogrammierungSchreiben <Datenverarbeitung>Elektronische PublikationForcingHilfesystemKonfiguration <Informatik>InformationMetropolitan area networkLaufzeitfehlerAnalytische FortsetzungTorusAdditionFahne <Mathematik>Interface <Schaltung>FlächeninhaltRippen <Informatik>RechenwerkVersionsverwaltungMultiplikationsoperatorDemoszene <Programmierung>Familie <Mathematik>CodeSpezifisches VolumenDivisionVerkehrsinformationGenerator <Informatik>HomepageSuite <Programmpaket>Projektive EbeneDatenbankTermProgrammierungDebuggingMailing-ListeStochastische MatrixQuellcodeSkriptspracheFront-End <Software>Äußere Algebra eines ModulsWrapper <Programmierung>System FProgrammpaketNabel <Mathematik>StichprobenfehlerNegative ZahlDifferenteKlasse <Mathematik>Funktionale ProgrammierungBildschirmmaskeEinsFramework <Informatik>Motion CapturingSichtenkonzeptProgrammierumgebungURLBrowserKonfigurationsraumComputeranimation
32:37
Gleitendes MittelCOMGoogolRechenwerkEmpirische VerteilungsfunktionTopologieSchaltwerkMomentenproblemHochdruckStatistischer TestMAPKernel <Informatik>Web-SeiteSichtenkonzeptLesezeichen <Internet>Elektronische PublikationSpezielle unitäre GruppeURNSummierbarkeitBildschirmfensterMetropolitan area networkVarianzFächer <Mathematik>OvalGamecontrollerSuite <Programmpaket>FehlermeldungLaufzeitfehlerMinkowski-MetrikImplementierungCASE <Informatik>SystemaufrufBitAggregatzustandKernel <Informatik>MakrobefehlSignifikanztestProgrammierungPatch <Software>ZählenInterface <Schaltung>System FApp <Programm>Projektive EbeneProtokoll <Datenverarbeitungssystem>DifferenteVerzeichnisdienstAutomatische HandlungsplanungFunktion <Mathematik>MomentenproblemTragbarer PersonalcomputerCodeVerzweigendes ProgrammVirtuelle MaschineDeterminanteBildgebendes VerfahrenParserComputerarchitekturÄhnlichkeitsgeometrieHochdruckFunktionale ProgrammierungRegulärer GraphStatistischer TestKomponententestDateiverwaltungService providerElektronische Publikationt-TestLeistung <Physik>LaufzeitsystemProgrammbibliothekQuellcodeKonfigurationsraumSystemprogrammierungVariableKategorie <Mathematik>MetadatenProzess <Informatik>Kontinuierliche IntegrationARM <Computerarchitektur>WhiteboardMultiplikationsoperatorEreignishorizontGebäude <Mathematik>Endliche ModelltheorieLie-GruppeMereologieGesetz <Physik>Feasibility-StudieOrdnung <Mathematik>Einfache GenauigkeitQuick-SortFormation <Mathematik>IntegralStrömungsrichtungEinsEntscheidungstheorieDigitalisierungElektronisches ForumDelisches ProblemTouchscreenBitrateRichtungServerTermRechter WinkelTreiber <Programm>FrequenzRechenwerkProdukt <Mathematik>PunktCodierungComputeranimation
41:36
ProgrammSignifikanztestDifferenzengleichungSuite <Programmpaket>Statistischer TestStichprobeElektronische PublikationSichtenkonzeptLesezeichen <Internet>BildschirmfensterDreiMailing-ListeWikiCOMGoogolMarketinginformationssystemMaximum-Entropie-MethodeSpeicherabzugQuick-SortElektronische PublikationTurnier <Mathematik>SignifikanztestRankingStatistischer TestVerzweigendes ProgrammComputerarchitekturHypermediaFigurierte ZahlProgrammiergerätSuite <Programmpaket>VerzeichnisdienstWikiProgrammierungProgrammfehlerComputeranimation
42:52
BildschirmfensterCOMMailing-ListeSoftwareGoogolARM <Computerarchitektur>Elektronischer DatenaustauschMinkowski-MetrikZweiComputerarchitekturStatistischer TestSystemzusammenbruchFunktionale ProgrammierungLineare RegressionKonditionszahlSystemprogrammierungSystemplattformSystemprogrammKernel <Informatik>Analytische FortsetzungTragbarer PersonalcomputerÜberlagerung <Mathematik>Suite <Programmpaket>Elektronische PublikationQuick-SortModulare ProgrammierungProgrammierumgebungSignifikanztestCASE <Informatik>EmulatorProdukt <Mathematik>E-MailUnrundheitProjektive EbeneRechter WinkelHardwareTreiber <Programm>Reelle ZahlDateiverwaltungWurzel <Mathematik>Schreib-Lese-KopfMereologieMomentenproblemIndexberechnungMultiplikationsoperatorAggregatzustandDatensichtgerätInverser LimesWasserdampftafelKonfiguration <Informatik>IdentitätsverwaltungArithmetisches MittelComputeranimation
Transkript: Englisch(automatisch erzeugt)
00:00
Work? Yes? Lights off or on? What's better? Whatever, this way.
00:23
OK, so let's get started. Hello, everyone. My name is Julio Marino, and I've been talking today about a project that we started at the end of last year, which is titled, as you can see here, the PrevisD Test Suite.
00:41
First of all, thank you for coming. There's more people here than I thought, especially considering that they put this other presentation next to us for LibreSSL. I expected many people to go there instead. I think it's unfortunate because, well, the LibreSSL project is a very hot topic these days.
01:00
And also, it's unfortunate because if they had had tests, maybe they wouldn't have had these problems they had. So they should be here instead. Anyway, so we have 45 minutes. I'll be talking for about 40, maybe less. And I'll try to leave at the end some time for questions.
01:21
You can interrupt me if you want as well. But otherwise, just keep them for the end, and we can discuss them later. So if you are going to forget everything I say today, at the very least, please try to remember this. You are here, and I'm here because we are building PrevisD, and we want PrevisD
01:41
to be a production quality OS, something that people can rely on. And well, having a test suite is something that is just a tool, right, something that helps us achieve this goal. And well, the test suite already exists. It's not something that we are going to do. It's something that already exists. And I'm going to discuss, well, what have we done
02:04
and what we will do. So before getting into the talk, so I want to explain to you who am I, because I'm quite new in the PrevisD project.
02:25
Let's say I became involved in VSD around the 2000s, and I became a NetVSD developer in 2002. And since then, I've done many things in NetVSD. I started working in packages. I maintained GNOME for a long time.
02:43
Then I got involved as part of some other code. I wrote TempFS. And when I did TempFS, I knew nothing about the kernel. So I was very lost in what I was doing, and I had to write code. And at the same time, I was writing some manual tests so I could make sure that the code I was writing
03:01
worked and kept working. And this was quite frustrating because there was zero help in the base system to write these kind of test programs. I just wrote my shell scripts. They kind of more or less run, but it was all very ugly. And when something failed, I didn't know why, et cetera. So that was in 2005. And then in 2007, I did another summer of code project
03:22
in which I started working in the test suite for NetVSD. And that's when the ATF project started. And it basically comes from there, right? So I tried to provide something that more or less followed the same workflow that I used and that could also
03:41
use most of the tests that we already had in the tree. Move a bit forward, and then in 2010, I realized that the ATF code base was not very nice, and it was very hard to maintain and extend to some things I wanted to do. So as an experiment, I started writing it
04:01
in a different project called Cuba, about which I will talk later. And well, you can think about Cuba as the replacement for ATF, and we will see about this later. So then comes 2013, when I attended EuroBSDCon, and I talked to people, and they
04:20
said we are very interested in the test suite, and we want tests for FreeBSD, and blah, blah, blah. And it seemed like a very interesting opportunity to just port what I had done from NetBSD to FreeBSD. And that's more or less how I got involved in FreeBSD. Then soon after, I got the commit bit, and I've been working on this project since then. That's on the open source side.
04:42
So at work, I work for Google. I'm a site reliability engineer there. I work in the storage infrastructure team. And everything I'm going to present here is not sponsored by Google, let's say. It's something I do on my free time, or on my 20% time at work, as you may know.
05:03
But unfortunately, we don't use BSD there. So this is just my fun project. OK, so what are we going to see today? So first of all, I'll give you a brief overview of the FreeBSD test suite,
05:24
just what the goals are, and how you can use it, and how it's more or less organized. Then I'll give you another view of CUA, basically what's the internal structure, how the command line works, how you can write some tests,
05:40
which we'll see with some code samples. And then I'll just glance over some of the future plans, and then we'll wrap up. 40 minutes, there's not much time for anything, so I'll just quickly go over these things. And if you have any questions about, you want to add more details, just ask me later or email me later, and we can talk about in much more detail.
06:02
OK, one thing I'm not going to do is, I'm not going to cover testing fundamentals in this talk, something I've done in all the other talks about test suites, because I'm assuming you already know this, otherwise you wouldn't be here. I think I don't need to convince you that tests are important.
06:20
So yeah, I'll skip that. And with that, let's get started with the test suite. So why do you want a test suite? Well, I already covered this before, right? At least in my case, I'm not previously writing a toy. I want to write something that people can rely on, something that can run in data centers, right? Something that people, or especially companies,
06:41
can just use and take advantage of. Something that they can be proud of saying, hey, we're using FreeBSD because it's stable, it's performant, it's reliable, et cetera, et cetera, et cetera. And I think many of us want the same thing, but I'm not sure if it's the case for everyone, but still. It doesn't mean that the test suite is only useful
07:01
for production environment. It's also useful for your average desktop or embedded devices or whatever you have, right? It's important to have it for all the cases. But the idea is just let's ship something that has quality, and let's make sure that these qualities can be demonstrated and it remains the same over time.
07:22
As you may know, there is a lot of interest in the test suite. You have probably seen this in previous FreeBSD Foundation talks, or, I don't know, the projects that they wanted to fund. Well, that's also the reason why I'm doing it, right? It's a way to choose something
07:41
that people will eventually use, and that's always interesting. Now, also, why is it interesting to have a test suite in FreeBSD? And I think, let's see how this is. We have an advantage over, for example, Linux,
08:01
in the sense that because the whole operating system is in one single source suite, we can very easily provide a comprehensive test suite that covers everything in the base system, including the kernel, user space, utilities, libraries, et cetera, right? You can test everything very easily
08:21
and make sure that the whole package works just because the development model of FreeBSD or actually all the BSDs. That's, I think, a big advantage, and I've seen some attempts to have test suites for Linux out there, and I'm not sure how well they work because it's this kind of external thing that some people develop,
08:41
and then you have to match it with specific versions and make it work, and I don't wanna, I don't know, it seems like a mess. Anyway, so that's the why, so the goals. The first goal of the test suite in FreeBSD is that it should run out of the box, right? The idea here is that we are developing a test suite
09:01
that can be used for both developers and for users. Usually, when you think about tests, it's something that the developers, when they write their code, they run, and it passes, so I can ship my code to the user, and that's it, but in this case, we also want the users to be able to run this test
09:21
because then, themselves, they can see and witness that the software works as intended, and also, they can be assured that this remains the case over time, right? You upgrade your machine, you can run the test suite again, and you still see things working, which, if it's only for developers, you cannot do that,
09:41
or you buy a new hardware server, and you install for a BSD unit, you can run the test there to see if everything is stable or not, or everything works as expected. I think this is a particularly strong reason for having it out of the box, because it allows us to do this,
10:02
and I'm not sure if anyone else is doing it this way. Actually, many people consider the tests a very important asset, and they don't distribute them outside of whatever they do, because, well, you have the code, and if you modify it, well, you have no idea of knowing what you are doing. But in our case, it's open source, right, so we can ship everything together.
10:21
Now, the second goal is simplicity, and the reason this is a goal, I mean, it's always nice to have simple things, right? But there are two things here. The first is that we have to be able to run the test, again, as I said before, out of the box,
10:42
without having to set up any VMs, without having to really understand how the test suite is built or anything, right? The goal was always to have just one single command that you run, and you can run all the tests in your machine, and that's it. And the other, the other goal, specifically for the FreeBSD test suite,
11:02
which was not the case for NetBSD test suite, is that I wanted to be able to use all the code for existing tests that was in the tree in the test suite without having to modify them a lot. If you have investigated the contents of the source tree in FreeBSD, you may have noticed that there are a couple of directories,
11:20
like tools slash regression and tools slash test, which include a bunch of tests, but there really is no framework to run them. And many of those tests are actually useful. If you run them today, you will find that either they can be broken, or they can actually find problems in the system, and that has been the case, right?
11:41
Those tests already existed, but because nobody run them, nobody knew that the bugs already existed. So that was a goal, be able to plug these tests into the test suite instead of having to throw them away. And the third goal for the FreeBSD test suite, which is more of a personal goal than of a project goal,
12:01
is to be able to experiment with new ideas. And this comes from the fact that it can be ATF to QR transition, that I was kind of mentioning before. So in NetBSD, we are currently using ATF, and I've been trying to migrate over to QR for a while, but there are some shortcomings that need to be addressed,
12:23
and I haven't had the motivation to do so, so we're stuck, and because I have motivation, there is no progress, and then things diverge, and it's a mess. In FreeBSD, because we are not tied to the old approach, we can just right away dive into the new approach without having to worry about backwards compatibility, and that's kind of motivating for myself
12:42
to just be able to try new things and see how they work. And eventually, if they work well, I'll be able to go back to NetBSD and say, hey, they are doing it this way, maybe we should do it the same way to avoid diversions, which is always nice. Mostly, that's something I didn't mention before, but
13:03
what's the ATF and QA, yes, the test suite is not the case, but ATF and QA are third-party packages, I developed them outside of BSD, and they run in Linux, they used to run in Solaris, I don't know if that's the case anymore, but there's no reason they wouldn't, and they run in BSD, of course.
13:22
Okay, so now let's get into how the test suite looks like. Okay, so you have a FreeBSD system, and you have enabled the test suite, which we will discuss later. You will end up with a user test directory,
13:40
which contains the test suite, that's all. If you wanna get rid of it, just delete the directory, if you wanna install it, well, you just need to make sure that that's there. This directory contains test programs, which are here represented by these blue boxes at the bottom, and it also contains a description of what's in the test suite,
14:02
and what the properties of those test programs are. This comes from these QA files here, that are more or less like make files. And they basically describe, again, the structure of the test suite.
14:20
Couple of things here. The first is that the layout of this directory is supposed to match the layout of the user source directory, and that's why we have their bin, and for example, user.bin instead of user.bin, so that developers can very quickly know where the tests for a piece of software are.
14:40
And that's also a difference between, with the NetBSC test suite, in which this is slightly different, they are trying to move to this model, but they are not there yet. But we've made a mistake at the beginning, we tried something different, it didn't work out very well. So this is what developers want, and I think now that it's a better model.
15:02
Important things here also are, well, we have directories, right? So every directory can contain one or more test programs, or zero, but there's no restriction for having only one. And that's the case, for example, for sed, or many more other things we have in the tree.
15:22
The other thing is that these legacy test names here, I just put them down, because in this case, they represent test programs that have just been migrated from the old test that we had in the tree, without having to touch them at all. They've just been plugged into the test suite, and they just run. And last thing to mention here is,
15:41
what do these files do? I can't answer it before, but basically, it's just, for these layers up here, it's the basically list in which directory is in which to recurse, right? And then at this level here, they say, okay, in this directory, I have this test program, and it's written in, well, sorry, it says,
16:03
maybe this program requires Perl, so the description can say that. So then the runtime engine that we will cover later, will be able to tell you that, hey, you need Perl for this test, but you don't have it, so I'm skipping this test. Or this test requires to be run in a PowerPC, or who knows what. There are many, many different options.
16:24
All right, so I said this comes, this mimics the layout of the source tree, but where does this come from? Well, obviously from the source tree. This is built from user source, as part of make build world. And you can note that it's exactly the same layout
16:44
as before, with the difference that we have all these tests, directories, interposed, everywhere that there is some artifact of the test suite, right? So if you're looking for where this file came from, for tests bin date, you have to go to
17:04
source bin date tests, and then this, right? This is very obvious for the leaves of the tree, but it's kind of confusing in the middle. And I'm not sure I like it a lot, but it's the easiest approach that we had with the current make files in FreeBSD.
17:22
We may change it later, but for now it's working pretty well. Also, this is a difference with the netBSD test suite. So in netBSD, the decision we made before was to put the tests under user source tests, and everything would be there.
17:40
And it worked well, but people, developers, don't realize that the tests exist, right? You go, you modify bin date, and you forget that the tests exist because you don't see them right there. And then when you build the tests and you run, sorry, when your continuous integration system runs the test, they fail. There have been requests in netBSD to merge the tests back into the corresponding directories
18:03
so that this is more obvious, just like we did with man pages many years ago, but I haven't done it there. So this is one of the new things in FreeBSD. Yes. You said that user tests is generated from user source. So the question is if user test is generated
18:21
from user source by some make command, and yes, so when you type make build world, or make world, or whatever it is in source, you get the output. All right. So this thing that you are seeing here in the test, in the tree, the good news is that it's already
18:40
in stable 10 and in head. So basically, if you want to start playing with this, all you have to do is enable the test suite, and that's it. The way you do this, you go to utc, slice, source.conf, you add with tests equals yes, and that's it.
19:01
You build your system as usual, you install it, and you get a test suite right there for you. In the future, I want this to be the default, but because we're still kind of cleaning up a couple of rough edges, it's not the case yet, but it will probably soon be. So you install the test suite, and when you build it,
19:21
you install it. So what do you do with it? You have to run it, of course. And the first thing you have to do to run the test suite is to install QA. Right? And the way you do this today is from the ports tree. This, again, is a difference from NetBSD. And I thought it would be shortcoming,
19:41
but I'm changing my mind. So the difference is, in NetBSD, we always had ATF in the base system, and we now have QA in the base system so that, really, when you run out of the box, you can execute your tests. But first of all, previously, we have this shiny new thing,
20:03
package-ng, which works very well with binary packages. So really, you can just do that command and get QA in a minute. You don't have to worry about anything. And the second is maintaining these things in the source tree, in the base system, sorry.
20:22
It's hard because of all the complex make files that you need to recreate. So the upstream version uses auto-make and auto-conf, and you cannot put that into the very system, right? So you need to rewrite the make files so that they build. And then maintaining that is complicated, and making sure you have the import new version is complicated, et cetera, et cetera. So keeping it in ports for now
20:43
allows at least me to be more agile in how we do things. And I'm not completely sure that leaving it out of the base system is a bad idea. The other thing is that you need to install QA from packages. But also, you also need to install many more things from packages because some tests are written in Perl or Python.
21:01
And if you want to run them, you will actually have to install those interpreters anyway. So you have to use ports like it or not, basically. OK, anyway, you have installed QA. How do you run the test suite? You just do this at the bottom, and that's it. After a few minutes, you will get the results,
21:21
and you will say, OK, everything passed or didn't pass. And you can later investigate why in various different ways that I will not go into in this talk because we will cover a lot of time. Right, the interesting thing about having just one command
21:41
like this is that you can now, for example, put that into Crony, if you like. Or you can run it after you change the kernel option or you upgrade your system. It really is just that simple, and that's the idea, right? You have to be able to run it frequently and continuously so that you get the most benefit out of it. So what happens when you do this?
22:04
And this ties to how QA internally works, right? Basically, this is the difference with what we used to have in ATF in that we have a database in the middle that splits the execution of tests from the generation of reports, right?
22:23
You run your tests with QA tests that will go over your user test hierarchy. It will store all the results. It will store all the output of all the test programs so you can later look at it. It will store stack traces if your program crashes, et cetera. And it will put it in a database that contains historical data.
22:42
And then with the QA report command, you can just generate various output formats, like text, HTML, and, since yesterday, JUnit, which I'll go into later. And the original idea for this was
23:02
to have the database wrapped by an RPC server. So you could just have many different nodes pushing results into your central test server. But it was the original idea, and given things I will explain in the future plans,
23:20
I'm not sure that makes sense anymore. But I need to think about it a bit more. Anyway, so we have generated a report, right? And the user format, voila. How does it look like? I'm just going to show you a very simple example from the HTML output.
23:42
And this is just a capture of the browser when I went into one of the continuous testing machines that we have. You can see the URL ql3.nyi.org. If you go there, you will see an index page with various runs, and you can click on one of them, and you will see something like this. So you just have, basically, a summary of what happened, and then all the test cases classified by their status.
24:03
And if you click on one of them, you will see some of the errors, some of the output. You will see the environment that we're running, et cetera. OK. So if you just want to see it yourself, of course, you can just go there again.
24:21
And you can do that now if you like. And that's basically where the test suite for FreeBSD today exposes to developers and users. And that's where some people go and see if things are kept stable. Now let's focus a little bit more on CUA, especially on the command line, so that you can navigate it,
24:41
if you like, basically. So here is how the command line structure looks like. If you have ever used anything like Subversion, Git, GPart,
25:00
or many different command line utilities, even CBS, these will look very familiar, right? You put your form name, any options that apply to all the commands, then you have to give a command, and then optional flags to it, and optional arguments, depending on what the command is. If you are ever stuck, you can use the help command
25:23
that you have in the command line. Or you can look at the manual pages for every subcommand. The manual pages are more comprehensive than the built-in help, because the built-in help only provides you a tiny summary of what every option and subcommand does. And the manual page will give you way more details
25:42
on how things work. If they are inconsistent, that's a bug, of course. So what commands do we have available for this thing here? There really are a bunch of them. And I will not go into details. But basically, we have a bunch of commands that are generic.
26:03
They allow you to get, I don't know, information about the program with about or help. They let you inspect what the configuration of CUA is at the moment, after the configuration file has been parsed. You can run SQL commands directly in the database. You can upgrade an old database to the new schema.
26:22
You can also, then you have the workspace commands, which interact with the current directory. There's a flag to change that, but by default, it's the current directory. And this is like tests. As we saw before, to run your tests, list to investigate what's in the test suite and what properties each test has, or debug, which allows you to run one single test
26:46
with the ability to debug it. At the moment, this only means that you run the test. There is no database involved. And all that there is is that you get access to the standard output and the standard error right away. And you can very easily read what's happening.
27:01
Ideally, at some point, we could just, for example, make this avoid forking so that you can attach a debugger easily. Or we could make it so that it attaches a debugger for you when things crash, et cetera. That's the long term plan, but it's not done yet. At the moment, it's just standard error, standard output. And what else do we have? Well, the reporting commands.
27:21
Report, as I mentioned before, and report HTML, which has some additional flags. And I'm not sure if it will stay there. But the one that allows you to generate multi-page HTML reports. Anyway, this is simple enough. And where things get interesting is how QL works internally.
27:45
So that's where the back end's coming. And that's a new feature as of maybe two years ago. So originally, QL was a project that intended to replace the ATF runtime
28:04
tools with something better without touching the ATF libraries. I didn't want to rewrite the libraries because they are decent enough in some cases. And the problems were mostly in the tools that you use to run them. So when I started QL, the only thing you could do
28:22
was to run ATF tests. And that was fine. But then I realized that there's all these people that want to get their simple test programs that don't use any libraries into the test suite, or they want to get test suites like, I don't know, the GCC test suite and plug it into our netBSD
28:41
or the previous test suite by rewriting it. And of course, you cannot expect everyone to rewrite their code to use the ATF libraries. So what I did was add support to QL so you can run tests of different frameworks, let's say. And that's where the back ends came in, right?
29:02
Again, for ATF, the C and shell libraries are decent enough. I'm not that happy with the C++ one, especially because there are many alternatives out there. Also, if we ever want to write tests, say, in Python, it makes no sense to write our own library because pyunit is very good.
29:21
So let's make use of everything that's out there. And we are trying to reinvent it unless we have a need for it. And yes, so then basically we have in QL we can run different test programs. We currently have these three back ends, which I'll go into a bit more detail now. But we could very easily write others. Actually, somebody emailed me the other day
29:40
with a back end to run pyunit tests. We'll see where that goes. So let's get into the details of the plain interface. And I'll just glance over the stuff very quickly. Here we have an extract of this file here.
30:02
That's in your source tree if you want to look at it, which contains a sample test program. If you go there, there is a C program and there is a shell program as well that you can look at. They are very heavily commented. And there is a make file also that you can look at. There is a QFI you can look at. Everything is there for you to just copy paste into your own directory if you want to use it.
30:22
But basically, what's a plain test program? These are the legacy programs we saw before. And it's basically an executable that returns zero if everything is correct, or non-zero if there is any problem along the way. As you can see, this has no structure in the sense
30:45
that from the point of view of the caller, this is a single program. Even if we have here three test cases, the caller doesn't know that. So if this one decides to fail, it will just exit. And the other ones will not run, which is probably the big deal for many things.
31:02
But when you want to get to more complex things, really you need tests for all your functions and class methods, et cetera, then you really want some additional control over how this works. First, so that you can, from the outside, say I just want to run this class test or something. And second, so that, I don't know,
31:22
if this test here decides to corrupt memory, you don't want this one to fail, because that would be a false negative. False negative, yes. So we want some kind of isolation between them. But this interface allows us to plug the simple test cases. And it allows also us to plug things like, again,
31:44
external test suites. Some other frameworks out there, when you run them and prove, I think it is, for example, they will exit 0 or 1, depending on what happens. So you can very easily write a wrapper script for that that returns 0 or 1, and call it from CUA, and you can get the results out there.
32:04
An interesting thing also is that even if this is very simple, from the CUA file and from the Makefile in FreeBSD, you can specify properties for these test programs, right? So if I happen to write this in Perl, because Perl is a base system, I can, from the Makefile,
32:21
say, there is this test program that requires Perl. And that will be registered into the test suite. And when you run it, you get a warning saying, hey, you need Perl, please install it. Or you need to run these in an i386 machine, or whatever you want. OK, so we're going to make this a bit more complicated.
32:40
And we'll do that with the ATF interface, which, again, you have the full source code is in that directory. The only thing we change is obviously ATF inside the plane. And the main difference for using this library is that in a single test program, you can have more than one test case, which
33:02
are defined by these snippets for these macros. And the runtime engine will ensure that each of these test cases is run in an isolated manner. That's in a different directory, so you can create temporary files if you have any problems.
33:21
It will create a different subprocess for you. So if you corrupt memory or crash, the others will run. You can't provide metadata properties for the different test cases independently. So you don't have a test case, one of these saying, I need to be run as root, or I need to be run as a regular user, et cetera.
33:43
Importantly, also, the CLI for any test program that uses these libraries is exactly the same. So you don't have to know how to run a test program, which used to be a problem in the past, because they all expose the same interface. And you can look at the man page for one of them, and that's it, you know how everything works.
34:01
They also support configuration variables that can come from the runtime engine, and then you have various helper macros to simplify things. More importantly, as you can see, there is no calls to exit or rx or anything, because if you do that, you're not complying with the ATF interface.
34:22
But if you use those macros, which allow you to test for different things, then those communicate with the runtime engine, saying, hey, something failed here, with this specific line, with this error message, and please look at it. This is all documented in the man pages for ATF, so I'm not going to go into details of the macros,
34:41
but basically, again, the same structure, but allowing more fungal control over every independent test case. And then finally, we have the tab backend, which is something that appeared at the end of last year, when I started working in the FreeBSD test suite. And the reason was, I thought the plain, so first of all,
35:03
many of the old test programs in FreeBSD use output, the tab format, and I thought that would just work fine with the plain interface, but that's not the case, because these programs always return success, even if they fail. So I had to write a different parser that
35:22
would understand the output that these programs print on the screen, and determine success or failure based on those prints that the program does. But that's the only difference. So if you have a tab test program, it will also work using this.
35:40
Tab stands for Test Anything Protocol. It was originally a Perl thing, but now it's used by other stuff, I believe, I don't really know. And if you run tests that are released in FreeBSD that use this, you will notice that many of them are broken, just because of that stupid print app at the beginning. Maintaining that count by hand is ugly,
36:03
and in many cases, it's out of date with what the test program is doing. So in many cases, if you just fix that little thing in the existing test, they will just work. OK, so future plans. Kernel testing.
36:21
So everything I've said so far is for, you would say, user space, right? I didn't talk about the kernel at all. And there is a lot of interest in having tests for the kernel. And so this is what we have to test the kernel today, which is basically nothing, unfortunately.
36:44
The thing is people from Iceland have been sending patches to the FreeBSD test suite to add tests for the kernel. But they have a problem that they just tweak some CCTL and then try something. And that's very ugly, because you're changing the global state of the system. If somebody else is doing something in that machine,
37:01
they will suffer from things that are unintended. So we need a better way to deal with kernel tests. And I do not have an answer for this, but I think that what we do in netBSD, which are RAMP kernels, is probably a nice way to do this, although it requires quite a bit of work.
37:23
So for those that don't know what a RAMP kernel is, RAMP is, or any kernel methodology, I don't know how to call it, it's something that allows you to get kernel code and build it in user space without modifying it.
37:42
So that you will, let's say, you have an implementation of the kernel API in user space. So you can take a modified kernel code, build it as a regular user program, and run it. So netBSD will use this for file systems, the TCP IP stack,
38:01
some pseudo devices, some other things. So you can basically just get their code, build it as a little program. You can run GDB against it. You can make it crash by injecting, I don't know, malicious file system image into the driver and see what happens. You can do unit testing of any function you want,
38:22
because after all, it's just a user space process. I think it would be very nice to have this in FreeBSD for testing the kernel. And if you're interested, let me know, we can talk. But I warn you that I think you cannot port what netBSD has done, because basically, this is just
38:41
rewriting the APIs that we have in the kernel. So I think for FreeBSD, we would need to do something similar. OK, so this is an open question. I don't have an answer. But what I do have answers for is what we need to do for test suite soon. And let's have more tests and support
39:00
more architectures in the machines that run tests. But adding more tests, that's something we can do either by getting tests from netBSD, or by getting them from a couple of project branches that come from, I think, Juniper and Isilon.
39:22
And they have plenty of tests that we could just import. Architectures, that's just for the machines we have running. At the moment, I only set up i386 and a and b64 test ones, but I'd love to have some others, especially ARM and PowerPC. And the more interesting thing that I want to do, and which I learned in the Dev Summit these last days,
39:42
is to merge these two things, right? So Jenkins, no, let me backtrack there. So I had a lot of plans for what QS would do in the future, especially support some of the continuous integration features that you wouldn't want.
40:01
But that's a lot of work, and I just learned that Jenkins does everything. I was interested in doing, and much more. So because I don't have time, and this is all my free time, so it makes sense to just reuse Jenkins. And over the last few days, we have been working on trying to tie these things together.
40:23
And there are two things to do here. The first is getting the Jenkins builds to actually run the tests. And that's not done yet, but it shouldn't be too hard, because we can probably reuse many of the code that we have in the testing machines at the moment. And the other thing is make sure that when QR runs tests,
40:42
the output of them is plugged into Jenkins. And the good news is that that's already done. Well, it's not anywhere. It's only my machine so far. But this is the output of the FreeBSD test suite, parsed by Jenkins. And with this interface, you can just go, you know,
41:01
we need the build, we run the test, you have all tests, ordered by directory. And here you have the things that failed. You can just click on any of them, you can organize things, very easy. And we can get a lot of power by reusing this very easily. We just need to finish plugging things together. And then I'll think if the long-term plans for QR
41:23
have to change at all to make the project more focused on only supporting the test suite and forgetting about all the continuous integration features, which may be a nice idea, but I still think a lot about it. Anyway, to conclude, things to remember,
41:42
everything I talk about today is already in current and in the stable branch. We will continue to merge things into 10 as much as possible so that at least we keep things, you know. Because there's a lot of churn now in tests, it's good to bring them into the branch so that when we fix bugs and we backport them,
42:00
we can make sure we also backport the tests. To enable the test suite, just add that setting to your source.conf file. To run it, that's the command you can run. This directory here has a bunch of examples for you to look at and learn. And the wiki page that I will show you now
42:20
has more details on how things work. And again, you do not have to learn the ATF APIs to write tests for the previous test suite. You can use very simple programs and just get them running very easily. And with that, any questions?
42:44
You said that everything that's currently getting tested is unusual. And on the other hand, you said that certain tests have particular architecture dependencies. And I was wondering if you knew why certain user space tests would not work on every architecture.
43:05
So the question is, I mentioned before, why some tests would only work, for example, in PowerPC when they are in user space. Yeah. So the obvious example is because some utilities
43:21
for user space may only be available in some platforms. At least in the case for NetBSD, you didn't have fdisk for PowerPC, right? You had something else. That could be the case. Also, if you write kernel tests, as I mentioned before, just like tweaking CCTLs and stuff,
43:42
you may still be doing things. I mean, it's not a nice kernel test, but people do them. And you may need to mark them specially. Fair enough. Thank you. Anything else? Yes.
44:01
Uh-huh. That's a good question. I don't know. I don't think, exactly. So there is people discussing this. And I think, I don't know if it has been announced anywhere, but they even chose a name for the project. And they are, I think if you look at the previous testing mailing list, you will see details about this performance
44:23
test suite. And it's really building something separate from the regression test suite. Because plugging it together may not work very well. Yes. So the question is, if you can be notified about specific test
44:51
failures. At the moment, for the continuous testing machines, that's not the case. There is nothing there to notify you about anything.
45:01
I'm pretty sure we can do it with Jenkins, but I don't know. But yeah, that's actually one of the goals. I want to be able to send emails to people when things fail.
45:26
Yes, yes. I mean, having you needing you to go and check, that's not going to work. I want to have the notification path, but I don't have it yet. But yes. How many others does that? Any more questions? Could be a goal or not.
45:52
You were saying that, well, you don't want to affect the running system, but is that, why is that a problem, necessarily? People can spin up VMs all over the place.
46:02
Maybe it is reasonable to say the kernel suite requires that you don't do this in the middle of the production system. Yeah. I mean, so the question is, why do we want to have a round if we could just dedicate the test suite for kernel and mark it as such? Well, it affects kernels in every kernel, often,
46:25
for sure. But I was thinking you could maybe make a kernel module. A kernel does the kernel module, and you think it's compiled as kernel module, kernel module gets loaded. Tests get called through some indication.
46:43
And that probably is a better test than if you did it with some sort of user space emulation kernel, because the kernel is so complex, you test the functions in their own environment. They might recur, and you put them in the mold that you're a kernel. Yeah. I mean, again, I don't really have an answer. There are many options.
47:01
And so you can have VMs, and you can spin a VM to run the whole test suite. So for example, what we do in that BSD, we have a test that exercises a race condition in the rename function. And it does that for all the supported file systems.
47:22
And it does that really quickly, right? And if the regression appears, then the kernel panics. So you can test all the file systems, all of them, even if one of them crashes along the way in a few seconds. You have to spin a VM for the whole thing. As soon as one crashes, you would just kill the whole thing. Or if you wanted one VM per test case,
47:42
it would be very costly. It's worth investigating. I really don't have an answer, right? Rump is just one of the many alternatives. And it doesn't cover all the cases. You still need something else if you want to test device drivers with real hardware. And that we should do as well, but I do not know. Mm-hmm.
48:02
Right. Anything else? OK. Then that's it. Thank you for coming.