Merken

Make your tests fail

Zitierlink des Filmsegments
Embed Code

Automatisierte Medienanalyse

Beta
Erkannte Entitäten
Sprachtranskript
it they after trying to fix the public it's like some the the problems that come to my early morning talk on I'll have how to make your test frame of mind
might have things at the example of ElasticSearch also the library was initiated was the where to to
tell you something about testing and elastic search I happen to be a developer at ElasticSearch GmbH apart from that I'm member of the pages of the Foundation cofounder of approaching the would hands up if you know much more want to the cofounder Bill investment if you need an next the wake that the Mr. technician compared the the microphone working or is there something we can change you have could something words but it's this thing about I think that what is fixed if you need an exit it if you need an excuse to talk to you have your employer paid for for a trip to Poland just go to notable less reptilian what about the less experienced as they do when search in general and France so let's
start with a few questions how many of you write tests I don't care which 1 no 1 would have hoped and in here regularly check coverage about how like regularly also means what we call the months once every year I don't care who they are and tests regularly the so
pretty much everything should be fine well all group tests are green everything in their minds funny little work
someone randomized testing was introduced in the scene so someone found there is something very odd if you called a function that generates an integer between integer minimum and maximum and so let just about compute see an absolute from that should always be positive right however it's exactly those there's this like 1 exception to the rule and that's where you get the minimum value so although everything you're tests and check looks like a fairly green world and looks like a well-shaped that some object that is what the all road actually looks like when we look at our kids we don't check all the corner cases and this is where our customers find box FIL how do we go
about just things those quantities I do have a little child if you've been here a year earlier this morning you have seen here and I don't have a lot of time to clean our flat every day or every week so we look into meeting all of these jurors was the automation was used by 1 of those who must every 95 and it starts its work every evening all I have to do is take the garbage out and this is what I like to do with testing as well let me
introduce you to carry such randomized testing the principle of also known as property-based testing this bound many often that means that is known under if you do have a hacking background you may be familiar with the term puzzling that down there's also similarity idea that play although you we don't try to specifically uncovered security holes when confronting the system of unexpected input but we just try to enlarge the phase of the expected value that we expect to expose the system to so let's take a look at
what heritage supplies of the framework to generate random numbers random strings with different properties that can be ASCII strings that can be UTF-8 strings can be you had its things specific them properties it can generate specific carries a characters from specific concepts it can go and generate different and environment settings like changing your LocalTalk uses the runs under like imagine running those URIs systems under development and on the attack and on the other continuous integration environment was a local but having had some of that actually lives in China owing entropy that have a different local funding and not finding do because you test them as wrong local setting so you don't run your test just was random input because then of course it has a reproducible in anymore more each test will be based on a reproducible specific the calling and the system will tell you that people you that that test threatened was when it failed for you can reproduce it on your machine you can repeat 1 test run multiple times makes sense especially if in the in the during the test run you generate different dataset so having you have the same on test set up the same lottery run with different input values it supports for test time so many it has happened to run to along the killer value it detects leaking but those in your test environment but also in your production code when tried out on Apache military cover 2 streets leaks just by running the tests you will in the period between from without changing anything you can't annotate tasks like image like remember those passing it has hit that were always very annoying because there are multiple minutes just booksy at likely annotation to and it will run them just to on the endocrine system or only when you start the test you to a specific and command line switch OK the online right test that of which I do not know the input it only makes sense for a certain type of tasks and 1 of those types of tests of mine actually checking the result was the person computing point is that the case imagine a function that book integer value it's very cheap to check that the values that you have are actually stored but it's reasonably costly to thought who what do is you implemented sorting algorithm in your test you generate a set of numbers and you run of sorting function and at the end to only checks your numbers of these have been sort of the the the and as a use case for randomized testing is when you know often algorithm that is slower than the 1 you implemented for real for production but that is simpler to implement so you don't have all the fun part of optimizations and there you put the slower growth in your randomized test you on that over you generated in good and make sure is that the output of the identical production and every checkers to if you can think of an algorithm that achieves that 2nd in a cheap very cheap way compute the upper and lower bound on the result imagine adding a and B as the number you can make you can make the assertion that the result of a plus B there shouldn't we be larger than either a or B it it's lower than obviously something is wrong so just especially for these kinds of tests that all you can define as an upper and lower bound on the result you although of course will make sure you have also have deterministic task was specific input and data to and check the output of the of function this will only give you some broader coverage in your input data so what does that look like after adding the character which randomized testing depend on the tree build wanted gives you is a annotation 8 which we see here below sea and test annotation repeat gives you a hint that this test should be repeated multiple times in this case iteration the 100 means that should be repeated 100 times and the mean the repeated 100 times each with a different input that input data why is it a different input data that because over here we generate a random integer value we can given just an upper bound on the to a given day lower and upper bounds can we also have a function to generate just any random integer without the use of about thing here when will this is very generate our data and we feel our list where the land short value here we don't have the balance between have them if you wanted to so s explained before we do the sorting and our sorting algorithms and in the end which of the checks that you and area thought so to summarize on the unit test level which are right input where's the fixed things that we can repeat our tests if that fails in order to increase the search space for pattern knockout encouraging continued integration we can rerun tests to cover as much as I data that SPQ as we want to and we need we can even specify to just run the iteration things in here and continuous integration and all the local box just run it on Monday OK so that the the unit at level and less expensive and
1 step further His so far we only looked at unit tests can recover even more of course you can cover even want to think about coding in Java out your client may run with the friend Chadians for instance involving a run the different JVM optimization parameters and if you have a
distributed system you may even want to vary the number of nodes that just the runs on you may want to vary the number of processes on each of these nodes in a lot of the tree various number for each 1 test integration test 1 the various a number of you don't know that that the error test cluster consists of an integration test and of course of J we all the very JVM optimization parametres we vary the Chadian that we have balance so what that black Tupelo's them all the new world and the ElasticSearch road was to uncover various Chadian box belt that lead to index data corruption that lead to precious that's that we now can give specific recommendations on which Fabian versions up to to the end of my integration we recommend the users to Grand ElasticSearch how
does it look like for the old that we have a bunch of
machines both on Amazon and I'm running in the cloud and smell or instrument hard metal boxes where we don't have a virtualization layer in between theories on each committed smoke test and therefore likely running for jet Java unit test there's a regular run off tests that go against the rest API we test on the set of on and not only on different parts but also on different operating systems there's I remember if I remember correctly this deviance is you would do this then since Windows probably by now even more we ran on different types of cloud instances like smaller ones larger ones so for which you have a decent coverage there to visualize again we we test again ElasticSearch core being the unit test level for the test against each of our API Java client API and because the gets a RESTful API with West class we do have a backwards compatibility version where we check out there a random by supposed to be backwards compatible with back backwards compatible region of elastic search and running those of In parallel and checking that they have both compatible no wouldn't pick
doesn't have 1 development culture actually impact pretty large and rich struggled quite a bit less stabilizing things in the past accused once you do have randomized testing if you have put it in a hedonistic test with a monistic include and didn't that humanist firemen what you can do you develop a check out the code run that straight and hopefully it except for travel versions being defended by some Dutch but if this test run green everything should be fine however here if you happen to have like 1 1 is a status that that you're a developer didn't check on his local box suddenly it will fail and continuous integration at some point in time so you need to have some kind of responsibility there to check what went wrong I don't think and continuous integration system and to find the developer who can fix that or have cultures that we were everyone looks very and everyone considers that 1st priority to stabilize the bill another impact sure you do have a decent test you'd right now but as soon as you into it I promise you as soon as you know I'm integrate testing in New York build you'd have suddenly will become very flaky and that's not due to the nature of randomized testing been the use of futures in the the fact that there probably are many back system right now that you're not aware of just to give you 2 anecdote I tried integrate I tried integrating randomized testing in would Chris trouble I ran into him and would have built on top of Hadoop and that has the integration tests had been wouldn't that's that they always wrote to the town directory in a very specific location that if he had to build random parallel with pressure another thing fermented was that the developers reliant upon armlock closings in the time that if you have many tests and then JVM running that was very nice so each randomized that were collect crashed for this rep leakage and that's not even introducing random data and a previous employer mine I indicate I played around with randomized that that I look for a class that looked reasonably easy to a little bit of she'll master computing distances and such I introduced only random data the how hard can it be used to find box forward code that's been running for some decades actually pretty easy and took 1 nite and we had like 2 or 3 cases where it fails while being the only data mining others being bounding boxes the wraparound that no 1 had thought about because the client after that we're using that particular piece of code never any interesting issue but it was just essentially a ticking time bomb bomb bomb until someone in the client and developing genes would have hit the problem that down there that too unfortunately it took weeks to iron out all boxes it is like if you if you have a rich fixed 1 of these issues in the fall of the tuple week and then we found another 1 another we we find another 1 and that's just because there were either inexperienced developers who didn't think of the boundary cases all animals developers who were happy to have just computation text and they know that works so they have like a psychological boundary to look for authorization and trust me I've got the technological boundary of all I like cats are green so at some point in the day I will stop searching people think that break FIL 4
if if you introduce random that thing what happen in yeah don't have the notion of the I frame and of commuter actually is the 1 who broke a bone because it could are small that some random generated data that approach of past Soviet you will need to have kind of someone look at that and dry object or you will have team would altogether depending on the team sizes may take quite a bit of some kind this this particulate can be helped if you have a randomized testing in place for long enough because at some point you will realized that it's easier to and deal with these issues if you have reproducible error thank you shouldn't have to have that rely rely too much on timing you shouldn't have tested rely too much on having all of the system up and running right now ElasticSearch rubric heavily toward moving away from having many integration tests to words having just a lot of unit tests those that the amount of coated could be broken and that could have caused the issue this being reduced and another thing you should establish is to teach developers have to write tests that are easy to reproduce and we've had many developers who founded very easy and very rewarding to do integration test because if you go to the last layer and he established an index of breeding our test testing framework and it's by design very easy to do that so that even customers and do it however it although and all the like and white developers to just right into the integration testing need to cheat you developers like really go down there to the unit testing theories and get that right although the land cover box and your environment in our case that uncovered many such cases in each of our environment each time oracle releases the new Chadian the pre-release version of being run again that at least was being but that can Elastic Search and quite often that we run into in into problems that at the end of the day nearly 2 next corruption issues and that in ElasticSearch will need data loss so we try to uncover the these it's an very early to
summarize randomized has been getting is not available it can cause a lot of problems here the 1st of all it can cause a lot of problems the kids if you introduce a to established code base it will uncover many books at 1st you will have fix on the other hand it also cont cannot replace test was deterministic data like we've tried to come up we've done that with that knowledge can be found from refactoring recently and we found many cases rare just having it bit a test was imminent dictators and a lot easier in order to check the salt and you may give you people coverage even because the checks can be more restrictive however the fundamental nature of the test gives you have a broader coverage so my recommendation would be to write additional unit tests and To address unless testing in order to cover the quantities you can add marginalization in order to cover inflation that out that what we are about to do right now like half the whole cycle from checking up to building a release and the packaging instead of installing the packaging and then running the tests against the simple version that really become full cycle and 1 1 note introducing randomized helping needs a lot of discipline for fixing if you look for instance at the and the or multiple bills that I've seen companies in house you really really need to go out and pick the best quickly because it is the here the fact that we run was random data what we end up with is a continuous integration testing system where the test become like figuring they go from green to red should being threatened that's not something that you want have that something where developers were very quickly and stopped looking at administered integration system at all the you wanted to have always green so that you can rely upon I checked and added and I didn't break anything this otherwise if it's sometimes we know sometimes when people will not the only need 1 and the other thing you want to know something about it ElasticSearch right now actually that we run in the global multiple instances of multiple operations are operating systems on multiple JVM possibly in parallel those sometimes you run into the following issue you have a reproducible issues and it failed on every test run so suddenly have separable effect that you in the morning you go to the office and you can build failures and all of them caused by the you really wanna cut out otherwise people thought filtering the the most because really really can deal with 10 to 20 month each state just want build issue that's very nice so if you've
looked at your watch some of the stuff of that at this talk of actually shorter than initiated or less than then as advertised on the schedule I do have a few T-shirts here and I do have a little bit of what's called to here that I been told each library much for thinking that I would like to ask you and any if you have any questions but I would like to hand out should for the best testing theories that I get like if you know my background at frost call ended up other companies will usually do it takes a microphone tended to everyone to tell them to tell me their background I have lots of the I'd make it a little bit different want years so we have a lot but of what question I mean as I see this is basically you help to fraud which is what your orbital recommended that falls from integration test because because that it's I think it's getting even more people because some of the systems are very complex and how would you tackled that I mean if they could you break all the time FIL adolescent search we do it at at the integration test level In our case it makes sense because customers running the system in different environments if you deploy going to the same environment was thing database and back all the time you don't want to write my theory that make any sense of unless you want to change the database at some point in in the near and future and argue that makes them because we shifted that library out there and people are free to run it on whatever instance for real computers against whatever Chadian was then was whatever it local evidence in how so we're actually run our integration tests with that you're right it does raise the likelihood of breaking on the other hand we although also support for customers FIL we'd rather have these great breakages in the integration test was then in a call that we need to on the within 2 days is the if these are you know much you don't have that much pressure to take such an integration can and you can spend more time analyzing blocks but in the end it was found to 2 things they make use of them easier to analyze when something really and B have long been that actually helps to fix system and in our case that even helps and what action because FIL same lots of that available on the boxes of customers and downstream users to an online I think further questions common authority get them from a technical standpoint we saw how it through so called let's say in the test data generators and we saw the annotation which costs the environment to repeat the test but said how so that how do you actually build a randomized test city it was completes the sample and I would know how to do it myself example of at that actually was completely because of all life on of the character and I think very rare and that gives you the all the annotations loss function and that 1 all the I said I if I remember correctly timestamp-based and that will generate the speech and based on that the that the data being generated I give it takes the it if you can extrapolate the API it already has function of generated awaiting random integer and that's essentially but also the space of so it's not random integers in the mathematical cryptographic sense of the word but it's really random as in we generate a bunch of integers this presentation get i'd like a teacher and I develop suffering musicians and housework therefore the Ministry of Social Development is so this is an anecdote and that the building a brand new beautification using Google beep toolkit and at the same time 1 of the managers had this great idea that we should also to tourists and basically into into and testing and so using a brand new system using we driver Concordia into the into EGY testing and it was I was a little bit skeptical then the beginning thinking how can this really be greater than all the integration testing doing all of unit testing as a so we're into heat and it was the truly at giant printed out and 1st we had all these random failures and this went on for maybe 4 or 5 6 months so good at developing for body yeah and for 6 months were getting these random various because selenium responding to fast and of test verifications wouldn't work what it was like a hundred different problems with that and another team was put professional automated testing team was put to our side to help us fix all the problems in the in the desert and after about 6 months we basically the interval warts and because it made of the project was of features had to be developed really quickly and lots of box and basically the into intestines saved or sites and finding all these defects and it's time Scott but in the most trees for the id but my experience and that's what I want to ask you in the into is it was really only possible because management support to supported the whole idea without management and accepting that developed speed enormous amounts of time trying to debug defects and or efforts to try to keep the more hot because developers with that by well it's held without reasons 3 times that of the different look at it anymore I was an enormous investment and without the backing of management it wouldn't have been possible is that is it and true and your expenses know it's through the analysis we get the backing mostly because I like to think clearly and having backs analysis so that actually increases the amount of money we spend on support so doing this kind of testing upfront say something about a lot of money in the end but you're right that developers spend an enormous amount of time fixing these issues finding these issues findings of developer can fix them and you can't do that without management I ate it work for other companies before where word was a management decision to yes we know that such and such configuration can generated by also someone be down the degrees will work it's all management positions that we don't fix it so in the in that an environment establishing the integration testing was randomised input probably doesn't make a whole lot of sense if you don't have the on the idea that you want a fix it in it will only make you build more unstable any of the story but the we yeah in how and so on just a recent case so in our integration tests we here are did sign says that's trenches 0 is creating of some entities so through Jenkins so it's a trend of threats all the time to do on our own doesn't machines it was green so and 2 uh that it showed us that some In Jenkins we we we used like I'm not some not not really integrated at environment because we took all edges hyper is going to to create entities and term determine the driver was found uh dealing with some of the creating of of the sequences for the sequence annotation and some uh it's was stripped out than uh the the creating of the sequence in our environment so we had to like come Oracle driver so the local Oracle driver
bound to that took the took the annotations Results parameters and enter on all machines it's the created of the the into to somehow but it to squelch was like umm by about about it all the time and so it helped us to it's it's it's it's was just some it is you with so with other environments that showed us that the 2 we have we we have to to to use different uh take a look and different should drive us to to see that the software can fail at because of that because the driver work differently so it just a recent issue we we're we're we have heads of this case commonly uh that's so yes the integration tests show show a show us that uh you really have to have to look on the on the sum didn't different kinds of environments to this the what so 1 must story medicinal project we have some we have also some testing pain in different areas want borders which is more the easier part as I'm the only 1 develop on a Windows machine and everyone else the stability omega west so we had a lot of issues of these and environmental stuff um so the easier 1 was 5 separators um while we had also was so it was not configured to use a lot of proper line ending so we will calculating the hash on the file and it worked on on a Linux machine but the comparison fail on on on Windows because we had a different Goddard different line separator on and off line on the Windows machine and the more complicated tests are and we also the as synchronous test on on the actor system we use so it's they were working was time modes of some seconds and on the continuous integration these tests start to fail when other tests are running because the CPU is to loaded so we also get very flaky tests and this is quite annoying so long you think you can't cover it was randomized testing that small problem of test environments and scheduling they are they running in the right order so that actually we had similar to the problem of small writing like we had configured our test to around with the range of nodes the velocity which nodes and wait for some of the problems of the columns of the actually were too small to post all these notes on them that they were running and you also have different time another issue we ran and you that the more vulnerable than trying out new environment and those of course we're generating all sorts of errors because they were not yet that but the arrows meant to the development mailing list so every developer look like 0 my god something's wrong not not the most trying about and indelible from that them out of T-shirts but is still to review which you can squeeze the question about these turns of the plastic this you have these very violent advocate other like I got a few questions from what I've talked a lot about the randomized testing frameworks that you use how many of you are Java developers there was 1 not His this being no similar frameworks of your favorite of programming language anyone know what really how many of you open source developers so many none of the developments that have so I hope you get infected so 1 thing I have seen in the imagined community is there so they take practice but each patch that get your as being automatically run applied to the system around the tests against and you get an automated report how many of you do that 1 2 3 can tell me the project you will and of the In instance of that you 1 was for all of them but can you take place tells more about the implications of that have a good impact on how many pages you get stored helpful I think that yeah it's not 1 of them so we call on the group you because you think the implemented I think there was someone out there in the process your our group all but all all the things who that have no implications or something you know strategy in the more of this the of the the and when and where there was also think you of the whole so of of all and I have all the thank you the for all the so what manual and so I've heard about it only last week anyone here heard about mutation testing this what your experience the time more on the closing more on the flaws in search of a use the goal programming language and Demetrius's as far uh go follows and all moralist beginning to define Bunsen burner has made up yeah yeah it's it's super easy to use but I'll go totally crushable so that be assessed there was someone else over here I don't remember the face it this is complicated to retain it you have to obstacle for imports through didn't the generated to and the them dishes histotype Baker what was so the problem and what crossed my last that is the and 1 last question when we have all the environment in we started doing it mostly many many only just to get up and running and then we had 1 1 1 instance was that of the operating the similar instance for another operating system 1 went and 1 of the JVM right now we are at a point where doing all also this manually doesn't work anymore how many of you went through the trouble of stealing your existing infrastructure in EWS and ended up using something like a puppet show whatever what have you lower yeah someone who is the is where work and that the Bank of New Zealand and there's a team called the default Skopje surely you know you as well and they sort of take care of that kind of thing they use proper to stand up and runs fossil costs a development hints and not entirely sure how exactly it works we make requests and say hey we noticed a moment in the fuel stated so yeah but it seems to work really well if people know how to use public and I think that then take them they'd only a few days to get used to it but it seems to be the way to go and comparing that to the posterior you would wait for weeks and weeks and somewhat with a few hours really felt like came you don't have any more questions that have a life that you can go and have a copy of this town there's also going to
Signifikanztest
Offene Menge
Rahmenproblem
Signifikanztest
Computeranimation
Homepage
System F
Software
Freeware
Programmbibliothek
Wort <Informatik>
Elastische Deformation
Softwareentwickler
Softwaretest
Gruppenkeim
Signifikanztest
Codierung
Computeranimation
Softwaretest
Signifikanztest
Lineares Funktional
Quader
Extrempunkt
Applet
Schlussregel
Ausnahmebehandlung
Demoszene <Programmierung>
Objekt <Kategorie>
Funktion <Mathematik>
Rechter Winkel
Ganze Zahl
Ein-Ausgabe
Vorlesung/Konferenz
Resultante
Komponententest
Punkt
Euler-Winkel
Minimierung
Iteration
Oval
Raum-Zeit
Computeranimation
Gebundener Zustand
Übergang
Netzwerktopologie
Resampling
Metropolitan area network
Algorithmus
Einheit <Mathematik>
Geschlossenes System
Mustersprache
Randomisierung
Vorlesung/Konferenz
Hacker
Phasenumwandlung
Funktion <Mathematik>
Softwaretest
Signifikanztest
Umwandlungsenthalpie
Lineares Funktional
Kategorie <Mathematik>
Personalcomputer
Ähnlichkeitsgeometrie
Nummerung
Ein-Ausgabe
Biprodukt
Arithmetisches Mittel
Generator <Informatik>
Menge
Ganze Zahl
Ordnung <Mathematik>
Zeichenkette
Subtraktion
Quader
Term
Framework <Informatik>
Überlagerung <Mathematik>
Task
Virtuelle Maschine
Leck
Multiplikation
Datentyp
Softwareentwickler
Bildgebendes Verfahren
Universal product code
Kontinuierliche Integration
Mailing-Liste
Physikalisches System
Quick-Sort
Kreisbogen
Zufallsgenerator
Integral
Summengleichung
Uniforme Struktur
Array <Informatik>
Flächeninhalt
Softwareschwachstelle
Mereologie
Signifikanztest
Parametersystem
Geschlossenes System
Subtraktion
Komponententest
Prozess <Physik>
Quader
Minimierung
Versionsverwaltung
Nummerung
Physikalisches System
Computeranimation
Integral
Summengleichung
Netzwerktopologie
Client
Knotenmenge
Automatische Indexierung
Vorlesung/Konferenz
Parametrische Erregung
Fehlermeldung
Instantiierung
Subtraktion
Komponententest
Quader
Virtualisierung
Applet
Klasse <Mathematik>
Versionsverwaltung
Physikalische Theorie
Hinterlegungsverfahren <Kryptologie>
Eins
Übergang
Virtuelle Maschine
Client
Regulärer Graph
Geschlossenes System
Datentyp
Bildschirmfenster
Vorlesung/Konferenz
Elastische Deformation
Parallele Schnittstelle
Differenzenrechnung
Softwaretest
NP-hartes Problem
Signifikanztest
REST <Informatik>
Mereologie
Speicherabzug
Streuungsdiagramm
Instantiierung
Einfügungsdämpfung
Bit
Komponententest
Punkt
Quader
Natürliche Zahl
Klasse <Mathematik>
Versionsverwaltung
n-Tupel
Abgeschlossene Menge
Framework <Informatik>
Physikalische Theorie
Code
Überlagerung <Mathematik>
Data Mining
Algorithmische Zahlentheorie
Client
Geschlossenes System
Endogene Variable
Randomisierung
Kontrollstruktur
Elastische Deformation
Abstand
Softwareentwickler
Parallele Schnittstelle
Signifikanztest
Softwaretest
Autorisierung
Green-Funktion
Gebäude <Mathematik>
Kontinuierliche Integration
Wiederkehrender Zustand
Physikalisches System
Kommutator <Quantentheorie>
Integral
Objekt <Kategorie>
Druckverlauf
Automatische Indexierung
Rechter Winkel
Wort <Informatik>
Computerunterstützte Übersetzung
Verzeichnisdienst
Fehlermeldung
Orakel <Informatik>
Randverteilung
Einfügungsdämpfung
Bit
Komponententest
Natürliche Zahl
Versionsverwaltung
Schreiben <Datenverarbeitung>
Raum-Zeit
Übergang
Resampling
Netzwerktopologie
Geschlossenes System
Vorzeichen <Mathematik>
Randomisierung
Kontrollstruktur
Vorlesung/Konferenz
Signifikanztest
Softwaretest
Addition
Nichtlinearer Operator
Lineares Funktional
Datenhaltung
Gebäude <Mathematik>
Stellenring
Systemaufruf
Ein-Ausgabe
Entscheidungstheorie
Scheduling
Generator <Informatik>
Druckverlauf
Twitter <Softwareplattform>
BEEP
Ganze Zahl
Rechter Winkel
Projektive Ebene
Ordnung <Mathematik>
Aggregatzustand
Instantiierung
Subtraktion
Folge <Mathematik>
Web Site
Quader
Gruppenoperation
Sprachsynthese
Kombinatorische Gruppentheorie
Term
Physikalische Theorie
Code
Algorithmische Zahlentheorie
Virtuelle Maschine
Multiplikation
Reelle Zahl
Stichprobenumfang
Programmbibliothek
Softwareentwickler
Konfigurationsraum
Analysis
Autorisierung
Soundverarbeitung
Likelihood-Funktion
Kontinuierliche Integration
Programmverifikation
Physikalisches System
Integral
Office-Paket
Minimalgrad
Druckertreiber
Dreiecksfreier Graph
Identitätsverwaltung
Wort <Informatik>
Resultante
Offene Menge
Gewichtete Summe
Prozess <Physik>
Punkt
Momentenproblem
Applet
Gruppenkeim
Computeranimation
Homepage
Freeware
Geschlossenes System
Bildschirmfenster
Vorlesung/Konferenz
A-posteriori-Wahrscheinlichkeit
E-Mail
Default
Gerade
Signifikanztest
Softwaretest
ATM
Parametersystem
Scheduling
Software
Grundsätze ordnungsmäßiger Datenverarbeitung
Strategisches Spiel
Projektive Ebene
Programmierumgebung
Fehlermeldung
Instantiierung
Geschwindigkeit
Subtraktion
Stabilitätstheorie <Logik>
Existenzaussage
Zentraleinheit
Framework <Informatik>
Virtuelle Maschine
Knotenmenge
Spannweite <Stochastik>
Lesezeichen <Internet>
Software
Netzbetriebssystem
Hash-Algorithmus
Vererbungshierarchie
Zeitrichtung
Softwareentwickler
Schreib-Lese-Kopf
Trennungsaxiom
Programmiersprache
Videospiel
Open Source
Kontinuierliche Integration
Mailing-Liste
Paarvergleich
Physikalisches System
Elektronische Publikation
Quick-Sort
Integral
Patch <Software>
Druckertreiber
Flächeninhalt
Mereologie
Eigentliche Abbildung
Verkehrsinformation

Metadaten

Formale Metadaten

Titel Make your tests fail
Serientitel FrOSCon 2015
Teil 47
Anzahl der Teile 80
Autor Drost-Fromm, Isabel
Lizenz CC-Namensnennung 3.0 Unported:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen.
DOI 10.5446/19583
Herausgeber Free and Open Source software Conference (FrOSCon) e.V.
Erscheinungsjahr 2015
Sprache Englisch

Inhaltliche Metadaten

Fachgebiet Informatik
Abstract It's easy as pie: before checking in, your test suite should always be green. Or should it? What if your tests are all green but you forgot to check one important edge case? What if your underlying system environment lets you down, but only under rare conditions that you didn't cover in your tests? This talk introduces randomised testing as used by projects like Apache Lucene and Elasticsearch based on the Carrotsearch Randomised Testing framework. It has helped uncover (and ultimately fix) a huge number of bugs not only in these project’s source code, but also in the JVM itself which those projects rely on. Writing unit and integration tests can be tricky: assumptions about your code may not always be true as any number of "this should never happen" log entries in production systems show. When implementing a system that will be integrated in all sorts of expected, unexpected, and outright weird ways by downstream users, testing all possible code paths, configurations and deployment environments gets complicated. With the Carrotsearch Randomised Testing framework, projects like Apache Lucene and Elasticsearch have introduced a new level to their unit and integration tests. Input values are no longer statically pre-defined but are generated based on developer defined constraints, meaning The test suite is no longer re-run with a static set of input data each time. Instead, every continuous integration run adds to the search space covered. Though generated at random, tests are still reproducible as all configurations are based on specific test seeds that can be used to re-run the test with the exact same configuration. Add to this randomising the runtime environment by executing tests with various JVM versions and configurations,and you are bound to find cases where your application runs into limitations and bugs in the JVM. This talk introduces randomised testing as a concept, shows examples of how the Carrotsearch Randomised Testing framework helps with making your test cases more interesting, and provides some insight into how randomising your execution environment can help save downstream users from surprises. All without putting too much strain on your continuous integration resources. Isabel Drost-Fromm

Ähnliche Filme

Loading...