Test ALL the Things – On improving the nixpkgs testing story
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 14 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/39619 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Year | 2017 |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
NixCon 201711 / 14
3
5
6
8
9
00:00
Statistical hypothesis testingStatistical hypothesis testingLibrary (computing)Function (mathematics)Execution unitScripting languageGastropod shellModul <Datentyp>Physical systemImplementationStatistical hypothesis testingModule (mathematics)Statistical hypothesis testingRight angleClient (computing)Hacker (term)Configuration spaceVirtual machineSoftware developerBitSet (mathematics)Moment (mathematics)Power (physics)Library (computing)Software maintenanceProgramming languagePhysical systemSynchronizationModule (mathematics)WritingFunctional (mathematics)Goodness of fitRobotUnit testingService (economics)ImplementationBuilding2 (number)Dot productServer (computing)Computer fileMetreExistenceBit rateJames Waddell Alexander IIWordTransportation theory (mathematics)Surface of revolutionConnected spaceFunctional programmingMathematicsOscillationProjective planeField (computer science)Computer animation
06:05
ImplementationDerivation (linguistics)Statistical hypothesis testingStatistical hypothesis testingStatistical hypothesis testingComputer clusterDerivation (linguistics)System callDefault (computer science)MassFitness functionMathematicsComputer animation
07:35
Computer virus2 (number)Order (biology)Derivation (linguistics)Statistical hypothesis testingComputer fileParameter (computer programming)Right angleFunctional programmingAuthorizationStatistical hypothesis testingGroup actionMathematicsAdventure gameLecture/Conference
09:11
Function (mathematics)Derivation (linguistics)SpacetimeComputer programmingHacker (term)Right angleImplementationNoise (electronics)TouchscreenInterior (topology)Parameter (computer programming)System callComputer animationLecture/Conference
10:24
Function (mathematics)Statistical hypothesis testingRight angleComputer fileDerivation (linguistics)Statistical hypothesis testingFunction (mathematics)Attribute grammarOperator (mathematics)Parameter (computer programming)Hand fanBEEPMusical ensembleSearch engine (computing)JSON
11:44
Function (mathematics)outputComputer fileBitFunctional programmingFunction (mathematics)Real numberData structureExpert systemComputer programmingMathematics
12:25
Statistical hypothesis testingRevision controlDefault (computer science)Computer virusDerivation (linguistics)Statistical hypothesis testingCASE <Informatik>Derivation (linguistics)Set (mathematics)Statistical hypothesis testingGastropod shellElement (mathematics)Electronic mailing listFunctional programmingProgrammer (hardware)Attribute grammarScripting languageSystem callSingle-precision floating-point formatRevision controlComputer programmingTouch typingFunction (mathematics)
14:15
Statistical hypothesis testingStatistical hypothesis testingComputer virusRevision controlDerivation (linguistics)Lie groupDefault (computer science)Surface of revolutionStatistical hypothesis testingPower (physics)Revision controlSurface of revolutionBuildingFerry CorstenINTEGRALComputer virusSoftware developerFunction (mathematics)WordWindowHand fanScripting languagePersonal identification numberStatistical hypothesis testingState of matter
16:23
Statistical hypothesis testingComputer virusStatistical hypothesis testingRegulärer Ausdruck <Textverarbeitung>Surface of revolutionRevision controlCode refactoringData managementMatching (graph theory)Message passingFunctional programmingStandard deviationMotion captureFunctional (mathematics)Regulärer Ausdruck <Textverarbeitung>BitString (computer science)Binary fileError messageNormal (geometry)Statistical hypothesis testingInterpolation
17:04
Function (mathematics)Computer virusRegulärer Ausdruck <Textverarbeitung>CodeDerivation (linguistics)Statistical hypothesis testingFunctional programmingProcess (computing)Error messageBuildingFunction (mathematics)Computer animation
17:46
Statistical hypothesis testingDerivation (linguistics)DivisorCodeScripting languageDistribution (mathematics)Statistical hypothesis testingStatistical hypothesis testingDerivation (linguistics)BuildingBitFunction (mathematics)Normal (geometry)Gastropod shellScripting languageLibrary (computing)Coefficient of determinationINTEGRALGrass (card game)Process (computing)Euler anglesFault-tolerant systemGroup actionData integrityServer (computing)Uniform resource locatorComputer animation
19:43
Revision controlStatistical hypothesis testingDifferent (Kate Ryan album)Statistical hypothesis testingoutputError messageStatistical hypothesis testingRevision controlBounded variationMusical ensembleLine (geometry)Distribution (mathematics)Set (mathematics)Different (Kate Ryan album)BuildingInsertion lossBeat (acoustics)Function (mathematics)Plug-in (computing)Parameter (computer programming)Extension (kinesiology)Endliche ModelltheorieCASE <Informatik>Functional programmingPresentation of a groupTheory of relativitySurface of revolutionMeeting/Interview
21:54
ImplementationStatistical hypothesis testingModule (mathematics)Statistical hypothesis testingStatistical hypothesis testingDerivation (linguistics)Attribute grammarMaxima and minimaElectronic mailing listRight angleAsynchronous Transfer ModeModulare ProgrammierungGoodness of fitData storage deviceWorkstation <Musikinstrument>Fatou-MengeGroup action
23:06
Statistical hypothesis testingImplementationModule (mathematics)Computer configurationDefault (computer science)Set (mathematics)Configuration spaceType theoryElectronic mailing listOffice suiteStatistical hypothesis testingStatistical hypothesis testingAttribute grammarBoolean algebraDifferent (Kate Ryan album)Address space2 (number)Computer animation
24:00
Statistical hypothesis testingImplementationStatistical hypothesis testingScripting languageInterface (computing)Statistical hypothesis testingGastropod shellBitPrimitive (album)Normal (geometry)Statistical hypothesis testingMathematicsMetropolitan area networkPower (physics)Hacker (term)Meeting/Interview
24:44
Execution unitData managementPower (physics)Digital photographySpecial unitary groupProjective planeData managementWater vaporPhysical systemLogicMathematicsSpacetimeType theoryHard disk driveModulare ProgrammierungStatistical hypothesis testingEnumerated type
25:44
Attribute grammarSpacetimeMeeting/Interview
26:38
Function (mathematics)Computer virusCodeStatistical hypothesis testingSystem callMusical ensembleMathematicsComputer programmingSpacetimeWeb 2.0Moment (mathematics)Extreme programmingError message2 (number)TransmissionskoeffizientProcess (computing)Analytic setMereologyExecution unitSymbol tableOperator (mathematics)Crash (computing)Revision controlINTEGRALInheritance (object-oriented programming)Statistical hypothesis testingDomain nameProjective planeDerivation (linguistics)QuicksortStandard deviationType theoryMultiplication signLibrary (computing)Scripting languageSoftware maintenanceGoodness of fitRight angleCASE <Informatik>Arithmetic meanPoint (geometry)MassKeyboard shortcutLine (geometry)BuildingLink (knot theory)Phase transitionFunction (mathematics)Semantics (computer science)Group actionComputer animationJSONMeeting/Interview
Transcript: English(auto-generated)
00:09
All right, so let me present to you Prof. Butch, who is a bar value and hacker and Haskell developer.
00:23
I hope you had a good nutrition phase. We're going to talk about the power to revolutionize next packages. And of course, of course I'm talking about testing.
00:43
Right? Yes, because we're all doing it. It's great. Everything's automated and we don't have any breakage ever on master. Okay, no, probably not. So what do we have right now? We have basically two sets of tests
01:02
in next packages at the moment. The first is in lib test. And it's testing, like the path says, it's testing the library. We have a lot of library functions, which are basically, because Nix is not,
01:21
not a general purpose language. We don't have as much as other languages, but we have quite a few. And we do unit tests for them. Not very many, but it's, I'd say it's okay. And the functions are very simple. You can run them while Nix builds.
01:41
The test released of Nix. And there's a second test set, which probably many of you know, which is the VM tests. These are quite impressive. If you don't know about them, they are basically, you give a module, you give like your system config for one or more machines.
02:01
And then you say, oh, here's a per script, which maybe waits for the machine and then connects to the other machine, runs a service and checks for the result, and then says, okay, or nah. And yeah, and those are pretty fast. You can start one in two to three seconds,
02:20
and the tests run for a few seconds, and then you have a complete test. For those of you who don't know the Quake test, which has two machines, a client and a server, the server runs Quake in server mode, starts the map, spawns a few bots,
02:41
waits for the player, the client machine, to join, waits until the client is fragged, and then makes a nice little screenshot, and that is then put into the out path, as far as I know. Maybe I could be wrong, and I could not even exist anymore. Those two sets, they share a problem, or they have a few problems that they share.
03:05
So lib slash tests, okay, it's maybe better than the Nix OS, the M tests, which are in slash Nix OS slash tests, and then there's a whole lot of files, but the real modules are spread throughout the Nix OS subtree.
03:23
So the tests at the moment are pretty far removed from the implementation, which, when you are writing a module, or fixing a module, you might not even know that there is a test, you might not know how to run the tests,
03:41
or you might change the module and submit something, someone merges it, and the tests fail, and nobody has any idea where they failed, what failed, maybe one or two maintainers know about it. That leads us to the second problem, they run separately.
04:00
So when you change the module, you have to start the test separately, and you have to run them manually, and all of these three problems basically lead to the synchronization of the tests and the modules, no tests being written for modules,
04:20
because nobody knows how to write them, or nobody knows that they even exist, and spit right off the tests, especially in the library functions, where you can change a library function, or add a new one, and then tests fail, and yeah. Of course, good CI could fix that problem, or meditate that problem,
04:41
and good tools could, like the bots that Graham proposed in the last talk, but maybe you can do better. I thought about the problem a bit, and I came up with two maxims, like what should always be ensured when you write tests,
05:03
or when you write testing systems, and the first one I came up with is, the test should be as close as possible to their implementation. If possible, in the same file, maybe right under the implementation, at least in the same directory,
05:21
very near, where you can always, when you open the folder, you see, oh, test dot. Like, when Golang does one thing very right, and that is every file has a test file right beneath it, right in the same folder, and you can't miss the tests.
05:41
And the second thing that is not done very often, it should not be possible to change the implementation without running the tests, and it should not be possible, like, to even run the implementation without first passing a test, which might sound a little bit far-fetched,
06:02
but bear with me, because it actually exists, and it works right now. For example, if you are in all packages dot nix, and you have the well-known hello world package, new hello, the default package,
06:22
and you want to test if it works without running it manually. Right now, the call looks like this. You have a call-to-call package, and it instantiates the package. I left the little hole in there,
06:41
because it should not even be possible to assign hello without running the tests first, so we should run some tests in between before it gets assigned. And of course, we don't want to recompile hello when some test changes,
07:01
because you added a test or something, so that's pretty much basic stuff. So the hello derivation, which is the right part, it depends on the successful build of, no, the left part, so the hello assignment, depends on the successful build of the hello derivation,
07:22
and some tests, whatever they may be, and however they get run. But the derivation mustn't depend on some tests, because if something depends on something else, you all know it, it will be rebuilt by Nix. We know such a function. Maybe there is a function in Nix,
07:42
which does basically that, it runs the left side, and it returns the right side when the left side is evaluated. Yeah, trace, it's implemented by which function?
08:01
Yeah, it's a built-in function, it has three letters, sec, right. If you come from Haskell, you know that function, and when you're in a functional setting, you sometimes want to have dependent order of evaluation, and then you set sec, which is sequential, maybe it comes from sequential, I think, and it evaluates the first argument
08:26
before the second, and then returns the second. So maybe we can just import it into sec, and it will work. Let's try calling sec with a new simple derivation which just writes the text to the file blah,
08:43
and then returns hello, and we get hello, okay, that seems fine. And then let's try it with the file foo. We change the left one to foo, and oh, crap. We get the same hello, but the tests have changed,
09:01
so maybe it should be a different hello, or maybe, yeah, we can see that hello clearly hasn't depended on the test. Oh, we are very sad. So could it be that there is maybe, there is a bit, and that's evaluating the derivation first, and then returning the second one.
09:23
Maybe, maybe, no. We even said no. But maybe we can do something about this. And so it happened that one fateful night last year,
09:40
Aslik and I were sitting in our hacker space, like two chairs apart, and it was very loud, and people were doing stuff, and I had my screen on the presenter, and was basically doing a pair programming session with Aslik over the noise of the hacker space. We were shouting between each other,
10:02
and I said maybe we can implement something like that, so then we can improve testing. It would be very much like sec, so we should call it derivation sec, and it has two arguments. The left one is elevated, the right one is returned, so the left one should be our test,
10:21
and the right one should be, the left one should be our test, and the right one should be our output derivation. So the idea that we had was maybe we can create a new derivation that is exactly like the output derivation,
10:44
but that runs the tests. So what should we do? We should first collect the outputs that exist of that old derivation, then we assign them to the outputs of our new derivation.
11:01
We get all the attributes, which are in there, out, address, and add them to pass-through, so they get passed through to the new derivation, and they will be inside as the derivation attributes of our new derivation, and now we still don't have the tests anywhere, so what do you do when you want to
11:21
depend on something inside a derivation? You just pass it as an argument, and it gets passed to the Bash builder or the builder script, and everything's fine with the word, so now we have a new derivation that clearly depends on our test. Of course, now we have all the metro attributes in place,
11:40
but there's still a file left, so we thought a bit about that, and then I just tried following. I took our old outputs and mapped over each output a Bash script, I did some shadow-scaping,
12:02
I resolved the link so that it points to a real file or directory, and then we submitted it into the next store, and it worked. We actually didn't think it would work, but now we had a complete copy of the old derivation, and basically the defsec function now works as intended.
12:26
We can expand it to a list. Functional programmer would maybe think, now we have a little, for a single element function, and now we just fold over the single element function on the list, and that creates a lot of symlinks
12:41
that we don't want, so the pragmatic programmer just turns it around, and the list version is the primitive, and the one element called is now the defsec function. And we are at the moment, and now we can implement our test function, which takes an attribute set of tests,
13:04
which are shell scripts with names, the red one, and the output, in this case hello, puts something into defsec L, which is a list of tests, and then the original output, each test will be put
13:21
like plain text into a script, out is touched, so that the build doesn't fail. If the script didn't fail, the build will not fail, and yeah, basic run common stuff, and we've got a test. To see that this works, Sekai Konnichiwa,
13:45
hello world, hello world, we use the new hello program, which we changed the derivation a little bit, so we put the original make derivation inside a let,
14:02
assign it to the name hello, oh, I reset my timer, okay, and call with tests, with our new tests. So, what could we test for new hello? Do you have any ideas?
14:22
Maybe, what's the most basic thing you can test for a binary? It runs and does not crash, right? How would you test new hello that it runs? It's just kind of simple, it's just executed and check the Excel status, or if you're really fancy,
14:43
you even check, like, you grab hello world, and yeah, and that's like a test script, and in a business world, you would call this an integration test for hello world, which is quite fancy.
15:04
You could go a little bit further and check the version output, and ooh, you can depend on nix values, which you previously defined in your let finding. So, we can now test if the hello world really is the hello world that it says it is,
15:20
or at least the developer didn't forget to change the version string, which already happened with some Python packages. And when you run that, you don't run it, you just say nix build minus a hello. You can see it's building the hello test, which gets KDL, then it's building the version test,
15:43
which gets VND, which are just shell scripts, that are then put into our new hello and run. As you can see, it first outputs hello world, and then the version number, those are grabbed, so if the grab doesn't work, it will exit one, and the test will fail, and you get a new hello.
16:08
This doesn't stop there, because that's just JavaScript executing, which you can do manually, but you now have the full power of nix at your disposal to bring revolution to testing.
16:22
If you look at the old, you can see the duplication. There's building hello twice, and then you grab twice, and so why not just capture it, refactor into a function, a function match standard out, which has an invocation and a regex,
16:41
and will have a nice error message if the regex doesn't work, and capture the bin hello part, and now the tests look like this, just two normal nix functions with a bit of string interpolation. What if we want to grab for goodbye said world instead?
17:05
This will happen. Hello test will try to get built. We get our nice error message from our helper function. Output did not match. It was goodbye cruel world. It should have been hello world. No, the other way around.
17:20
It should have been goodbye cruel world. It was hello world, and the builder for the test fails, and as you can see, the other test doesn't get rebuilt, and hello doesn't get rebuilt as well. So that's basically the idea behind
17:42
nix package testing. Package testing, not VM testing. You just need nix build process. I tried a few things, like what can you do with that? How can you integrate tests inside normal derivations? And the first thing was to eat my own dog food.
18:04
So we now have the testing functions. Why not test the testing function? For that, I quickly noticed that I needed to actually run nix inside the nix builds
18:20
to generate the output of the test, so I can grab the output of the nix run. So I thought, well, that's it. I cannot go any further. Then I grabbed a little bit inside nix packages, and oh, inside lib slash tests, they already have something like that to run the nix invocation, right?
18:44
So I factored it out into a shell script, and then I started testing the shell script first, because who knows if it is wrong or if it works, or doesn't work. So let's test that script, like run nix instantiate on a simple derivation,
19:01
then use that derivation, run nix build on that output, and see if it outputs what the derivation build script said. It should output and grab for that, and now we have guarantees that the nix invocation actually works, like that nix actually does what it should do.
19:21
So we basically did an integration test for nix. Sorry, I don't have scripts for that, outputs for that. But yeah, step five, that's a horrible joke, nobody here understands,
19:40
and I'm sorry for the two people that do. Yeah, another idea was, well, you have beats, which is a music tagger, and I could run beats in a test and do version checks. It's actually really nice for Python packages,
20:01
because maybe a lot of you know that Python, if it builds, it doesn't mean it actually runs, like most Python binaries just stop with an import error in the first line. So we could actually now, test every Python binary, for example,
20:22
for a version string, and automatically get guarantees for the whole Python set. I don't have it in this presentation, but what you can do is use the build Python function,
20:41
get in the argument, insert the with test function, and do a nice call, and then you can actually test functions for every input they get. And for every input that is different, it will run a different test, for example, a Python package. That could be done, it actually works.
21:03
No idea if we want that, but this is a nice thing that only happens in Nix, because you have a short variation phase, and yeah. So if you have the music tagger beats, we can actually go a little bit further,
21:23
and call a different version of beats inside our tests. And since we assign it above the three dots, after beats are the implementation of beats, then we have the full package with override capabilities, and we can override that full package inside the tests,
21:42
like enable a plugin we want to test, which is not enabled by default. That's one way, yeah. Another proposal I have, which goes towards the VM tests. Right now, as I said, the VM tests are very much in a different folder
22:03
than the actual modules, and so why don't we just integrate the test inside our modules? Keeping the first maxim I defined earlier, keep tests close to implementation.
22:20
These VM tests are, at the moment, run by Hydra. There's a list of tests which Hydra runs, and it's basically an attribute set which is nested in arbitrary deep nestings, with packages at the leaves. So if we could add arbitrary nested attribute sets
22:44
to the module system, then we can just add the test attribute. Then inside each module, you can set the test attribute to your foo test,
23:01
which is a derivation which runs a VM test, and in Hydra, oh, where's the Hydra part? And then in Hydra, it basically gets a config set and calls it config set. Don't know why it's missing. This is like a different testing approach than before,
23:23
so I left the package tests. Right, you can define the type, which takes like address off or list off, and the second attribute like int, bool, whatever,
23:42
and then you can say, oh, our test is a nested attribute of a nested type of attribute sets of packages, and yeah, that will do what you want, and you will get a nice list of tests for Hydra.
24:02
So I'm still actively working on this and have been for the last year or so on and off, and I want to hack on this at the Hackathon on Monday and Tuesday. First, the with test interface might get changed a bit, like having shell scripts might not be the best interface
24:24
for a generic interface. We should implement a few tests for packages, which are a bit more complicated than hello world. Then we should do a pull request with the primitives and the example packages, yeah, normal stuff.
24:47
Before I go, I have two more things I'm working on. I'm working on the tag union type for type system. For those of you who miss the some type way of doing type definitions,
25:01
you can now do something like that, which is like my hard drive has a power management value and it goes from zero to eight bits, and the zero is off, and 255 is something, and 200, and the first half is some value, and the second half is some value, and now we can do off or low from the low values
25:24
and or high from the high values, and you specify that, and yeah, that gets type checked. And I have a little project which does enumerators, small check like enumerators for next values, also with the module system to test changes
25:42
in the module system. All right, that's fast. Great idea, for the package tests,
26:03
where would you put the actual tests? Because for example, in the hello world, in the old packages, we have the hello world attribute. I guess you won't put it there, right? Because that would, the old packages would become too big.
26:21
So also I had the example earlier, right? Over here, I started with, no, no. In the first example, I had something like, yeah, the code package hello world, and then,
26:43
tests equals empty space, call package, and we don't want to put the tests here, because that's why I, in the actual example here, why I put this inside hello default. Oh, hello, okay, I missed that, yeah.
27:01
So we define the package in a lab binding and then use the politics. So we have this in the same file, which is as close as you can get. So how does this work with depending on tested packages? Since the output path changes each time, wouldn't that result? So if you add a test to some library,
27:22
wouldn't all the packages, depending on that library, need to get rebuilt, even though the library itself is still the same? If you test the library and add a test, yeah, that would probably happen. So you have that, but that's not something you...
27:41
That's always a problem if you have the tree and if you change something in the tree, which is the transits of the depending, where executables are depending. So you probably have to, still you have to create a leaf.
28:05
Which then tests the library. Or you can test the library and rebuild once and then test for everything that goes through the library. Yeah, interesting talk, thank you.
28:30
One thing that I would like to mention is that even though I find it fascinating, I also find it somewhat scary to be honest, because integration tests in general
28:41
tend to be a maintenance nightmare. And so anyone who has ever done or maintained integration tests for a non-trivial project maybe experienced that at some point. So in the end, if you have some non-trivial program, and well, hello obviously is a trivial one, and you have someone going overboard with tests,
29:01
like that funny little quake example, and then at some point in your super elaborate tests, which is like a couple of hundred of lines of bash code or whatever, something somehow goes wrong, because maybe some sort of semantics of the tool changed, and the package that you're testing might still be entirely okay,
29:21
but now you have to not only maintain your program, but also you elaborate tests, and I find that prospect actually scary, because in the end, you update something, maybe you wanna do a drive-by contribution and upload the version, and then something somewhere changes, and you have no idea why, and so I find that scary.
29:40
So I think that's mostly a problem of testing the wrong things. The only thing you want to test is things you absolutely want to be true, not things you think that might be true, or things that you think that might be true
30:02
that are true at the moment because the test passes, which is a bad thing to do, but guarantees you once your package is to give like, does not crash with an import error for every Python executable, thank you very much. The second part was, I lost the...
30:50
All right, that's probably the main problem with implementing that is making sure
31:03
the error messages fits the tests, that you don't get any empty lines when something crashes. And that's also a problem of getting good testing library support, where you have a testing library which has basic building blocks,
31:22
which then give good error messages. I just wanted to respond to that also, like it sounded like, I almost feel like characterizing it as integration tests is sort of blowing it out of proportion. I mean, these packages have tests of their own. So what we're actually testing is like a sanity smoke test
31:41
type thing of like, does this thing crash at startup? Like those are examples. Like if these things become complex, I think we're doing it wrong. The integration thing was mainly a joke. And to trigger some people, and I succeeded. Sorry. What is the problem with one program
32:01
and get stuff on the outside every evening? No, if it's a good program, it will always behave the same for the same inputs. Why don't we just write a test derivation besides package derivation and let Knox review built tests?
32:23
Is there a way to avoid rebuilding packages that depend on our package when the tests change? Do we mean libraries or? Well, you described how to wrap a package with a dummy package with some links at outputs.
32:42
What if we don't wrap the package with a test package, but add the tests package, for example, in our packages? Ah, right, you mean, there is already a test phase inside our make derivation standard end right now.
33:05
The problem with that is, these are good for tests that come from upstream. Like if you have a library which does tests, unit tests, and they get run, and if they fail, this is an upstream problem. But if we add integration tests,
33:21
we don't want to add them to the check phase, because then after our builds, the build will fail, and we have to restart the whole build. Well, I don't mean by check phase. I mean that what if we add another derivation, another package, for example, in our packages?
33:41
You mean like in other leave, like in other executable, which then gets collected and tested in- No. I want to exploit the fact that nox review rebuild.
34:11
I'm not sure. I wanted to rely on a nox review to run the tests. This way, we don't have to force Nix
34:21
to run the tests every time we build a package. Maybe that wasn't clear from the example. Well, when I run this, and when I run it the second time, it will always be a single operation,
34:42
because I don't have to rebuild the tests. Yes, it is. As with other derivations. I think what he wants to say is, why don't we just add a pass-through attribute, which doesn't change the derivation,
35:04
which isn't picked up actually while building, but only like with an extra tool that specifically runs the tests. That could be done, yes. That would like fix the rebuild case.
35:22
Yeah, it violates the thing I wanted to guarantee. As always, there's two, whatever your use case is. If it's higher up in the tree, then you have to be very careful what you change. If you add a space to a standard and bash script, you will trigger mass rebuilds.
35:44
You have to be very careful with testing as well. The higher up in the tree, the more conservative the tests have to be. If it's like a Python script nobody cares about, then just bash it with tests, okay, yeah.
36:01
Or intentionless tests, all right. All right, thank you. And maybe if Echo doesn't like this thing, for broch, for brach, for brach, then we could maybe add it as a build and derv sec.
36:24
All right, thank you.