Test ALL the Things – On improving the nixpkgs testing story

Video thumbnail (Frame 0) Video thumbnail (Frame 9115) Video thumbnail (Frame 11386) Video thumbnail (Frame 13769) Video thumbnail (Frame 15601) Video thumbnail (Frame 17593) Video thumbnail (Frame 18627) Video thumbnail (Frame 21374) Video thumbnail (Frame 24574) Video thumbnail (Frame 25594) Video thumbnail (Frame 26659) Video thumbnail (Frame 29586) Video thumbnail (Frame 32840) Video thumbnail (Frame 34640) Video thumbnail (Frame 35995) Video thumbnail (Frame 37095) Video thumbnail (Frame 38598) Video thumbnail (Frame 39945)
Video in TIB AV-Portal: Test ALL the Things – On improving the nixpkgs testing story

Formal Metadata

Title
Test ALL the Things – On improving the nixpkgs testing story
Title of Series
Author
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
2017
Language
English
Production Year
2017

Content Metadata

Subject Area
Abstract
While there is pretty good support for testing NixOS based services inside VMs, the same doesn’t hold for large parts of nixpkgs, especially for testing packages. I will present different approaches to improve testing. Over the last year I have been creating various testing environments, tried out different approaches and created tests for various parts of nixpkgs, notably package tests with very interesting properties. There are two kinds of tests in nixpkgs right now: /lib/tests: this is essentially a derivation that tests (some/most) library functions by checking their outputs. /nixos/tests: contains a list of tests for modules that use the VM testing infrastructure (`/nixos/tests/make-test.nix`). Both kinds of tests live in a different part of the codebase than the implementations they test, and have to be run separately, and manually (by hand or by CI). This leads to easy desynchronization of test- and actual implementation—the tests bit-rot over time. I therefore propose two maxims: All testing code should be as close to the implementation code as possible. Changing the implementation should not be possible without adjusting the testing code to make it pass. Coming from this angle, I recognized that a unique property of nix evaluation—short runtime—makes it possible to let nix packages depend on their tests. This is accomplished by a `seq` for derivations, discovered with @aszlig. I will present how drvSeq makes it possible to create package tests that are able to use the full power of the nix language for abstraction. It also does not cause unnecessary recompilation, like changing the checkPhase does right now. Going further, I will discuss best practices for including these tests into already existing packages. As well as packages, functions (and thereby whole language frameworks) can be tested as well, enabling a kind of testing that is automatically done to every usage of the function (i.e. every package in the framework). This could lead to interesting use cases. While VM tests already work with automatic CI support, their code location is not satisfying according to maxim 1. I will propose an integrations of VM tests as a module option, producing outputs neatly piecing into the current hydra setup.
Metre Server (computing) Existence Functional (mathematics) Implementation Scripting language Service (economics) Transportation theory (mathematics) Virtual machine Client (computing) Field (computer science) 2 (number) Statistical hypothesis testing Power (physics) Mathematics Goodness of fit Bit rate Functional programming Implementation Library (computing) Physical system Programming language Execution unit Projective plane Surface of revolution Oscillation Statistical hypothesis testing Statistical hypothesis testing Connected space Word Function (mathematics) James Waddell Alexander II Gastropod shell Modul <Datentyp> Physical system Library (computing)
Adventure game Mathematics Group action Computer virus Fitness function Authorization Mass Functional programming Implementation Statistical hypothesis testing Statistical hypothesis testing Derivation (linguistics)
Implementation Touchscreen Interior (topology) Function (mathematics) Parameter (computer programming) System call Computer programming Spacetime
Statistical hypothesis testing Expert system Function (mathematics) Parameter (computer programming) BEEP Computer programming Statistical hypothesis testing Hand fan Mathematics Search engine (computing) Function (mathematics) Operator (mathematics) Right angle Musical ensemble Functional programming Data structure output
Statistical hypothesis testing Scripting language Personal identification number Building Computer virus State of matter INTEGRAL Function (mathematics) Surface of revolution Statistical hypothesis testing Computer programming Statistical hypothesis testing Surface of revolution Hand fan Statistical hypothesis testing Power (physics) Derivation (linguistics) Word Lie group Touch typing Revision control Window Default (computer science)
Data management Standard deviation Message passing Computer virus Regulärer Ausdruck <Textverarbeitung> Matching (graph theory) Revision control Motion capture Functional programming Statistical hypothesis testing Statistical hypothesis testing Surface of revolution
Scripting language Statistical hypothesis testing Group action Server (computing) Computer virus Regulärer Ausdruck <Textverarbeitung> Scripting language Euler angles Code Function (mathematics) Grass (card game) Fault-tolerant system Statistical hypothesis testing Statistical hypothesis testing Derivation (linguistics) Uniform resource locator Process (computing) Function (mathematics) Divisor Functional programming Data integrity
Statistical hypothesis testing Beat (acoustics) Presentation of a group Group action Module (mathematics) Fatou-Menge Workstation <Musikinstrument> Parameter (computer programming) Function (mathematics) Modulare Programmierung Statistical hypothesis testing Revision control Goodness of fit Functional programming Endliche Modelltheorie Implementation Extension (kinesiology) Distribution (mathematics) Theory of relativity Data storage device Line (geometry) Surface of revolution Statistical hypothesis testing Personal digital assistant Revision control Different (Kate Ryan album) Musical ensemble Asynchronous Transfer Mode
Statistical hypothesis testing Type theory Module (mathematics) Computer configuration Electronic mailing list Office suite Implementation Statistical hypothesis testing Default (computer science)
Scripting language Execution unit Interface (computing) Projective plane Special unitary group Water vapor Statistical hypothesis testing Statistical hypothesis testing Power (physics) Statistical hypothesis testing Data management Mathematics Digital photography Data management Hacker (term) Logic Gastropod shell Implementation Metropolitan area network Physical system Spacetime
Function (mathematics) Spacetime
Statistical hypothesis testing INTEGRAL Execution unit Mereology Computer programming 2 (number) Statistical hypothesis testing Web 2.0 Revision control Mathematics Crash (computing) Operator (mathematics) Error message Domain name Computer virus Inheritance (object-oriented programming) Moment (mathematics) Projective plane Analytic set Extreme programming Symbol table Process (computing) Function (mathematics) Transmissionskoeffizient Musical ensemble Spacetime
so becomes nonsense Oscar de la hoya the only good zwischend face wedding cotton candy power to revolution as nix packages and course course in the word test in the tiebreak basic function basically cosmos of general purpose language english is back to the future many many of you know which is like a test of this press conference before the connection to this Machine service check mendoza city were almost constant in jute bags it ends transports seconds and villages completely test test which has to machine small servers of the server room craig in seven steps awards for the player of the client machines joint venture small the project and the ex-masters screenshots and then that potentiates some of an effective one existence ok now metal is because nothing os wm-test richie rich alexander hold of photonics osc should be relatively right now poor compensation which we knew what you and pixie lott you might not even out there is s did not know how to change it's up to sampling simon rattle fields nobody has any other way but what would you do not it was in the library functions way you can change and of course cubic meters of the problem last weekend weather mexico wyss be added value rating systems the change offers the implementation would like to see in the world and works
with funny july is to assess little fits in the picos with motives somehow possible to what a hello co-authors of the test so slowly stuck in with how I wanted so of course we do not want to recall halloweenfest change is pretty much basic so bright or fashion night party crashen masses
the basic as is now burning left side and return streetside walk fzj race is implemented by witch function now so picture the wrong way because of heart brain or wrong we want here in the wrong the setting youth home want to have fun or on action adventures corners change the authorities say hello to the test of change so what should be different hello baby can say hello kitty hesse has no test with can do
something about what happened at last sitting in outer space and touch screen our program so new softmaker space overlooked the implementation lag 6 as well as call it the nations at s arguments of the lift have been that all three goalkeeper rohde lab on chip biotest in the industry itself js
right watch foundation so that we can create music the operation bet black tests and videoservice collect the output exis operation if we have his too the operation wikipedia uk charts fits now the insight that the first national youth foundation is again d??sseldorf mister james abott new value to beep and sound installation economic policy arguments fits for example the fifth world innovation city fans by test of course we have the child abuse and place place search engine is keeping resort left
or joint structuring change as well complete copy of the basic function and no expert ends program
the output tests as well as kutlak plain text there is touch is supposed to be in the pale poison the script outrageously casual rankommen started as the works 6 conergy was halo hello world hello world program German and activation insider last what an increase helau helau helau no
simple is executed in jackson state is really window you even jack crabb hello world it meets in business vault cordes and integration test before halloween fans she will be from jack the virgin auto the pentax value switche grizzly the injured back autistic hello world rally is solarworld as it is not least developed to change the main Building diverse tests like andy Rogers scripts weather then kuttin to new health and run as you can see the world and develop said captain word excel and word on you Hi that's a stop the weekend but you know the power of revolution of the pin halo 2
because hugh grants manager skepticism defective
meanwhile function match standard outfit hessen the ok 60 helfer message of the way away is offenburg capture
function he is what goodbye cruel world and hello the hynix package testing package wm-test in sassnitz oz is also not
test is insightix bild.de generate the out of the hackett group the output on the Nothing like john tripp twelve tests teacher the autumn and immediately began even on the edge nothing in location the fact that agency script engine launches Testing business gives grass because you know how works plass castle so let's dance cryptics instanced to the simple the innovation for newstead nation was nix picture port and look and feel it out puts what's the patients with attitude output in grace location action action That was as well as data integration test of nothing a booklet's ball out 35 last half a job just because here is different than sam server to people do another as well
had like switches music maker on beats in the test version 66 rappen packages the costs with xtranews first top model import terror in the first line Extensions would urgently automatically All over Europe we would have that in the presentation but what i can do is just about the function would have the argument initiative test wrong actually test functions vw now a case like in potters different wants relative contest for example the package could be dead sea ??????works that is a revolution face of the world has since as it is now pizarro develops override valuable package insider tests equal a plug in wanted it builds between 2 opels yes apropos Kostorz wm tests first
wm-test a very much and different demanded the action mode so far and the integrated test kipping us first mexican julia there are claus trzeschan this as a test his team currently at hydro it is often transnet of basically averages, nest in a bid to sing with cars and storage short ccs then to develop the module system inside in inside idiot test maybe good station with swiss wm test in
best has a lightweight basically heretics config cent costs confixx need
that is then as before the enemy type which takes office and you'd like in whatever you can say yes the world one you want to get a list of tests in front of such a selective reebok in
the us and have been for the last resort to hack the apple hacker and man the interstate estenfeld change david pepping shell scripts offer best interface for general interface implement a few test woman packages complicated in the world if we already full power iq etfs an exact packages
for two more things are working on a working on a tec teenagers are photos on the eu mission has three fixed power management value and goes from zero to hero soft started trading too the first half is sun valley and that know very well what logic enduro values ??????and a half men values and space weather project the enormous raiders march it like it is nothing values ??????and changes in the water system
great example the empty space extraction
[Music]
here the problem change Silvia zu Creator left New York and can because just talking cure people zdf a fixed aging exclamation of symbols carried on to be because integration Tests in the Rocky Mountains Nightmare sunny cars ever done and education project so in the ndp has in the new studio program contains a jugo whirlpools tests like web analytics from a copy of super leverage tests which is like a prayer stay peszko dating something that wants to change you be ok just do you know me and you update with monitor drive-by could faster social and sometimes the changes is guaranteed in the moment the crash not import rfv repeater executive thank you very much second part wars job the domain domain my problem making the error message is that you Do not get testing [Music] [Music] [Music] and i do not respect the dead space inside xtreme leverage that tests unit This is an absolute problem meanwhile tests again and monsters too the checks finished why I thanked me touched replies clark of the example randes the transmitter n24 operation even deep sitting has the expected trainer josef cherfi change and space to standard version is guaranteed is may be like this that is spent at consumption when we become [Applause]
Feedback