Merken

What's the fuzz all about? Randomized data generation for robust unit testing

Zitierlink des Filmsegments
Embed Code

Automatisierte Medienanalyse

Beta
Erkannte Entitäten
Sprachtranskript
you will really sorry for the delay let's get started uh my name is more it's but I look at really in Germany and I'm going to talk about pretty much the same thing as tom just did and I hope those who watch also told you will find some interesting things 1st a little bit about me and what I want to talk about this topic um much of doing predictive analytics so that basically
means predicting the future and the actually it doesn't have anything to do with this it
looks more like this we have some machine learning pipeline uh with data
preprocessing and external data sources and and and some super-secret algorithms which produce and machine learning model and we work with big
data uh if you do an image search for big data you will find that the data is shiny and clean and most importantly from my experience the data is often not shiny and not clean um can be very complex and dirty and nasty to work with them so it's more like this but it's it's definitely so the big data is really a breeding ground for a we attach cases and things that can go wrong and of course we have to to find ways to protect ourselves against these things and discover potential problems before production and and randomized testing more quality dynamic testing is 1 such tool to help us find problems before going into production and to reduce stress for us and for our customers what OK so what do I mean by dynamic testing uh basically I mean both property-based testing and forcing them distance are things are usually seen as a kind of separate things which sometimes makes sense because they are often used for very different things but it's sometimes also said because there are lots of cool stuff in both fields and often it doesn't make sense to seperate them and so by dynamic testing I mean any testing where test cases are generated automatically for example by fussing which means the user gives some example input which is 10 mutated automatically or by parameter templates uh which means the user gives this summer generative model from which test cases are generated and then the function behavior is checked for properties back does not crash does not time would or and other universal properties like mathematical and expressions the function results which uh fulfilled In contrast to dynamic testing is of course that the traditional static testing where test cases are provided by the user and the function behavior is also precisely defined by the user but I want to compare the 2 types of testing bit and for this I want to introduce to at attributes of test the 1st this position which means whole closely the expected behavior of the function is defined and the other is case coverage uh which means that sometimes called input space coverage and it means what proportion of the input space is covered by test cases that had some discussion about them if case coverage matches so for example does it matter if we 5 true to the power of 64 cases or 5 thousand out of true to false 64 cases in any case it's still a really really small proportion of all possible test cases but I think it it does matter and I give you 1 example to maybe illustrated so let's say you will have you'll have implemented an algorithm and there's a numerical instability in your iris and that you don't know what and you don't know that there could be a numerical instability and only once 1 and thousand inputs are affected by this so if you use 5 test cases you have a probability of about 0 . 5 per cent to detect this instability if you have 5 thousand test cases you have a probability deal nearly 100 per cent of detectors and there are other reasons that like Tom before the independent called pass coverage can increase which is much stronger than just branch coverage so in general dynamic tests help you find classes k of test cases that you didn't think about before and usually it's like this static tests have very high precision or even a perfect position but no case coverage and dynamic tests sometimes positions because we have
to and define general properties but entire case coverage and steadily usually you can't have both high precision and high case coverage so the best way is often too would have just 1 of 1 at a time it makes sense to use both static and dynamic testing if you want to increase but you want to maximize robustness OK them with more practical um I'm going to show some examples for dynamic testing in Python and I'm going to use hypothesis which is which protects the testing for Python and I really like to work with the processes it's stable but still in steady development and it has a lot of innovative features show some of which I'm going to show OK 1st example so what your colleague went on holiday and he left you this function which calculates the Fibonacci numbers does anyone does everyone know what the Fibonacci numbers I think it's um OK so in any case but um maybe if you look at this called it's not really clear that this formula computes the Fibonacci numbers there the Fibonacci numbers are integers and here's some stuff with square roots and then an inverse here so we're not really sure if this is correct but our colleagues also leftist some tests is testing the base cases 1 and 2 is equal to 1 and and 2 more cases which are easy to compute in your head and even another case 50 50 which you probably looked up 1 more from from around this and everything is fine but I'm still not really trusting and this code so I'm going to write a dynamic test for this 1st set of hypothesis the most important piece of hypothesis from the user's perspective is that given decorator which provides a test function with automatically generated test cases and the other important pieces are strategies which define whole . ties generated so here we import a strategy for generating integers and then we use we import some settings stuff and tool dear randomized hypothesis because for the presentation I don't want any random behavior um and we limit the number of iterations and set the time out hypothesis has appeared in the example database where stores any examples so of that falsified an assertion before and every time you run your test you again these examples will be run to of course that makes the nondeterministic so and disabling this for for this presentation OK this is my dynamic test so let's look at this function 1st test recurrence takes 1 parameter and and we checked that the function fulfills the recurrence that defines the Fibonacci sequence combined with the base case and this recurrence should hold for all values and with at least 53 that's around this and we find that there is a forcing falsifying examples namely n equals 71 so this assertion fails for any was 71 let's take a closer look 1st at what happened so I to my edit the settings parameter to given and increased the paucity of the output and we see that the 1st falsifying example is an equal 805 and but hypothesis doesn't stop at this point it into its tries to find the simplest falsifying example no and for 1
for um and for integers this means finding the smallest counterexample so what it does and it starts with small values and equals 3 4 5 and then just pretty much of a stock as the binary search to find the smallest counterexample would give goes up to 67 here at which still works and then 131 which doesn't work anymore so it it goes back 1 step so it tries to find the counterexample somewhere between 67 and 100 and after a few more iterations of this is to find the example we have seen before and namely 71 OK to summarize this on the 1st step hypothesis runs sampling it generates test cases until it finds a falsifying example or until the maximum number of iterations is reached and when it's done when it finds a falsifying example it tries to to shrink this example of meaning it tries to find the simplest falsifying example for integers is the smallest example for strings this is the shortest string for example and we've seen troop he elements of hypothesis given which is the decorated supplies test functions with data and strategies which describe all data generated and I think we have seen that dynamic testing is very easy to implement and often very useful for mass heavy court where you have a nice beautiful properties that describe your functions behavior um but to some toy example with some less mathematically allergens called so this 1 so it's really just a toy example we want to test the quote function from the URL little package the called function prepares a string to be used in URL so you can see here it encodes the space with percentage 22 and we can run this static testing and passes true right the dynamic test for this and we have to think what do we expect from the port function what are properties that the called function should fulfill for any input um and 1 thing we might expect is that when quoting and then voting no information is lost
soul if you quote the string and then undevoted again you would expect to get the original string back and this writers and we find a falsifying samples if you find a bug in your sadly not or maybe not so that the um but anywhere we didn't know we just weren't careful about what kind of input we generate um the texts that strategy generates strings consisting of all possible unique characters and the counterexample we found here is just a control character so it doesn't make sense to encode this in your we can easily fix this 1 we import the string module and passed parameter alphabet to detect strategy that can tell you that we only want printable characters and now it seems to work fine OK and in general and handling impure objects like strings and they times can be a bit challenging not only in data generation but also in your assertions because of the underlying functions have a lot of special cases that you some money to handle um I just want to give you a quick overview of what kind of tools that are a little to assist you with this um hypothesis has building to help you for example the alphabets for string generation for 48 times there's the hypothesis daytime package and do you don't who knows what the fate factory package does talking to persons of the fake factory pictures really cool I think you can generate all kinds of of what kinds of every day fake dots like you can generate a random you are random for number as random e-mail addresses and hypothesis fake factory exposes use random generators to hypothesis so you can for example test functions steps taken you are a taker your Ellis apartment um hypothesis has also built in support for custom strategies so you can create your own strategies for example by mapping existing data types so the basic data types to your own custom data types Jack that was a quick and just to summarize dynamic tests increased case coverage at the cost of precision dynamic and static testing complement each other non is the great replacement of the order and sometimes it makes sense to use moles sometimes it's better to be more pragmatic to just use 1 of the for mass heavy called dynamic tests are really great really great um for if it gets more less mathematical more real words sometimes it can be difficult but there are tools available and steadily improving and if you have a problem with anything with any object and you think maybe there should be some tool available for this just you should just ask under hypothesis maybe for example because they want to know what OK yeah thanks for listening
and fact the at the end and then he
thank you very interesting topic maybe this is called by custom strategies by by other commissions for providing templates for for example protocol the not I think that doesn't exist yet that but Digital very interesting about come to mind that found in your production make yes at we had we had 1 combinatorial optimization was which would not only failed to look at as input it would take some distributions so that we predicted and it would only fail if there are 2 degenerate distributions and at least 1 non degenerate distribution in that case it would enter an infinite loop so that data you are expecting all could you can that be really hard about this is an example so it was that they see you're expecting if he can he Bahati examples might you've come up with that not that I think we would have never come up with the source so any more questions that a k please timing he means that you and I have that
Bit
Computeranimation
Arithmetisches Mittel
Virtuelle Maschine
Informationsmodellierung
Algorithmus
Varianz
Quellcode
Algorithmische Lerntheorie
Computeranimation
Resultante
Hydrostatik
Vektorpotenzial
Bit
Punkt
Prozess <Physik>
Gewichtete Summe
Iteration
Statistische Hypothese
Systemzusammenbruch
Template
Raum-Zeit
Statistische Hypothese
Computeranimation
PRINCE2
Statistischer Test
Arithmetischer Ausdruck
Algorithmus
Statistischer Test
Randomisierung
Funktionale Programmierung
Flächeninhalt
Kontrast <Statistik>
Wurzel <Mathematik>
Funktion <Mathematik>
Zustandsgleichung
Signifikanztest
Parametersystem
Fuzzy-Logik
Kategorie <Mathematik>
Datenhaltung
Template
Inverse
Ein-Ausgabe
Biprodukt
Datenfeld
Funktion <Mathematik>
Menge
Einheit <Mathematik>
Ganze Zahl
Ein-Ausgabe
Strategisches Spiel
Dynamischer Test
Kategorie <Mathematik>
Normalspannung
Message-Passing
Subtraktion
Klasse <Mathematik>
Zahlenbereich
Systemzusammenbruch
Kombinatorische Gruppentheorie
Signifikanztest
Code
Differenzengleichung
Ausdruck <Logik>
Quellcode
Informationsmodellierung
Fibonacci-Folge
Perspektive
Datentyp
Abstand
Speicher <Informatik>
Softwareentwickler
Grundraum
Ganze Funktion
Stochastische Abhängigkeit
Bildgebendes Verfahren
Fréchet-Algebra
Attributierte Grammatik
Leistung <Physik>
Schreib-Lese-Kopf
Trennungsaxiom
Algorithmus
Distributionenraum
Mathematik
Raum-Zeit
Verzweigendes Programm
Gasströmung
Überlagerung <Mathematik>
Objektklasse
Portscanner
Quadratzahl
Parametersystem
Attributierte Grammatik
Hilfesystem
Bit
Extrempunkt
Adressraum
Iteration
Statistische Hypothese
Element <Mathematik>
Template
Steuerwerk
Statistische Hypothese
Raum-Zeit
Lie-Gruppe
Computeranimation
Statistischer Test
Dedekind-Schnitt
Statistischer Test
Code
Randomisierung
Funktionale Programmierung
E-Mail
Signifikanztest
Parametersystem
URN
Schlüsselverwaltung
Kategorie <Mathematik>
Element <Gruppentheorie>
Stichprobe
Ruhmasse
Ein-Ausgabe
Hoax
Systemaufruf
Generator <Informatik>
Funktion <Mathematik>
Einheit <Mathematik>
Ganze Zahl
Strategisches Spiel
Dynamischer Test
Information
Cloud Computing
Ordnung <Mathematik>
Ext-Funktor
Zeichenkette
Objekt <Kategorie>
Zahlenbereich
Zeichenvorrat
Signifikanztest
Gegenbeispiel
Hydrostatik
Ganze Zahl
Stichprobenumfang
Datentyp
Distributionenraum
Diskretes System
Stichprobennahme
Mathematisierung
Gasströmung
Modul
Gerade
Programmfehler
Zufallsgenerator
Zeichenkette
Objekt <Kategorie>
Skalarprodukt
Parametersystem
Gamecontroller
Faktor <Algebra>
Wort <Informatik>
ASCII
Funktionale Programmiersprache
Distributionstheorie
Loop
Protokoll <Datenverarbeitungssystem>
Template
Strategisches Spiel
Quellcode
Kombinatorische Optimierung
Entartung <Mathematik>
Biprodukt
Ein-Ausgabe
Computeranimation
Unendlichkeit

Metadaten

Formale Metadaten

Titel What's the fuzz all about? Randomized data generation for robust unit testing
Serientitel EuroPython 2015
Teil 151
Anzahl der Teile 173
Autor Gronbach, Moritz
Lizenz CC-Namensnennung - keine kommerzielle Nutzung - Weitergabe unter gleichen Bedingungen 3.0 Unported:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen und nicht-kommerziellen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen und das Werk bzw. diesen Inhalt auch in veränderter Form nur unter den Bedingungen dieser Lizenz weitergeben
DOI 10.5446/20170
Herausgeber EuroPython
Erscheinungsjahr 2015
Sprache Englisch
Produktionsort Bilbao, Euskadi, Spain

Technische Metadaten

Dauer 20:15

Inhaltliche Metadaten

Fachgebiet Informatik
Abstract Moritz Gronbach - What's the fuzz all about? Randomized data generation for robust unit testing In static unit testing, the output of a function is compared to a precomputed result. Even though such unit tests may apparently cover all the code in a function, they might cover only a small subset of behaviours of the function. This potentially allows bugs such as heartbleed to stay undetected. Dynamic unit tests using fuzzing, which allows you to specify a data generation template, can make your test suite more robust. In this talk, we demonstrate fuzzing using the hypothesis library. Hypothesis is a Python library to automatically generate test data based on a template. Data is generated using a strategy. A strategy specifies how data is generated, and how falsifying examples can be simplified. Hypothesis provides strategies for Python's built-in data types, and is easily customizable.Since test data is generated automatically, we can not compare against pre-computed results. Instead, tests are usually done on invariants of functions. We give an overview of such invariants. Finally, we demonstrate how we use fuzzing to test machine learning algorithms at Blue Yonder.
Schlagwörter EuroPython Conference
EP 2015
EuroPython 2015

Ähnliche Filme

Loading...