Design Your Tests

Video in TIB AV-Portal: Design Your Tests

Formal Metadata

Design Your Tests
Title of Series
Part Number
Number of Parts
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date
Production Place

Content Metadata

Subject Area
Julian Berman - Design Your Tests While getting started testing often provides noticeable immediate improvement for any developer, it's often not until the realization that tests are things that need design to provide maximal benefit that developers begin to appreciate or even enjoy them. We'll investigate how building shallow, transparent layers for your tests makes for better failures, clearer tests, and quicker diagnoses. ----- * Life span of a test * 5 minute - why does this fail? * 5 day - what is this missing? * 5 week - do I have coverage for this? * 5 month - what's *not* causing this bug? * Transparent simplicity * one or two "iceberg" layers for meaning * Higher-order assertions - build collections of state that have meaning for the domain in the tests * bulk of the details are in the code itself * show an example * grouping for organization * Mixins * show an example * unittest issues * assertion/mixin clutter * setUp/tearDown tie grouping to the class layer or to inheritance via super * addCleanup * weak association / lookup-ability between code and its tests * package layout * other conventions * Alternative approaches * testtools' matchers * py.test `assert` magic
Keywords EuroPython Conference EP 2014 EuroPython 2014
Metropolitan area network Integrated development environment Internetworking Googol Code Line (geometry) Statistical hypothesis testing
Statistical hypothesis testing Suite (music) Context awareness Hoax Code INTEGRAL State of matter View (database) Multiplication sign Execution unit Source code Real-time operating system Function (mathematics) Disk read-and-write head Mereology Data dictionary Computer programming Mathematics Estimator Different (Kate Ryan album) Core dump Formal verification Set (mathematics) Fatou-Menge Electronic mailing list Unit testing Statistical hypothesis testing Type theory Message passing Arithmetic mean Process (computing) Exterior algebra Right angle Quicksort Writing Spacetime Point (geometry) PC Card Slide rule Server (computing) Functional (mathematics) Implementation Regular graph Inequality (mathematics) Protein Product (business) Statistical hypothesis testing Number Latent heat Natural number String (computer science) Representation (politics) Pairwise comparison Multiplication Information Key (cryptography) Cellular automaton Weight Density of states Cartesian coordinate system Software Object (grammar) Communications protocol Library (computing) Green computing
Statistical hypothesis testing Logical constant Email Suite (music) Building Greatest element Group action Code Interleaving Decision theory Multiplication sign Set (mathematics) In-Memory-Datenbank Data dictionary Mereology Perspective (visual) Coefficient of determination Mathematics Hypermedia Data conversion Email File format Moment (mathematics) Data storage device Ext functor Bit Web application Type theory Arithmetic mean Message passing Process (computing) Order (biology) Right angle Quicksort Metric system Slide rule Statistics Computer file Dependent and independent variables Online help Web browser Rule of inference Theory Number Statistical hypothesis testing Latent heat Root String (computer science) Hierarchy Operator (mathematics) Ideal (ethics) Condition number Data type Module (mathematics) Addition Pairwise comparison Dependent and independent variables Standard deviation Inheritance (object-oriented programming) Information Graph (mathematics) Content (media) Code Database Line (geometry) Cartesian coordinate system System call Software Personal digital assistant Mixed reality Blog Universe (mathematics) Object (grammar) Library (computing)
Statistical hypothesis testing Right angle Data structure
Statistical hypothesis testing Point (geometry) Group action Context awareness Divisor Code Multiplication sign Connectivity (graph theory) Range (statistics) Set (mathematics) Student's t-test Function (mathematics) Rule of inference Statistical hypothesis testing Coefficient of determination Different (Kate Ryan album) Videoconferencing Harmonic analysis Flag Endliche Modelltheorie Computer-assisted translation Metropolitan area network Pairwise comparison Multiplication Information Chemical equation Projective plane Gradient Unit testing Instance (computer science) Density of states Cartesian coordinate system Frame problem Arithmetic mean Word Personal digital assistant Logic Statement (computer science) Video game Summierbarkeit Right angle Quicksort Resultant Writing Local ring
everybody thinks that many of here so testing line this is me on the Internet environment many moles on the
last slide I think I'm talking afterwards so the magnetic but when advertising we do online betting on real to real-time bidding on online advertising always based on type production lots of starting document also program so but luckily this talk is fairly simple so simple that the core ideas that on a slide and a half which is basically that we know a lot of things are we think we know a lot of things about about what makes designing software and the good thing about this thing is that but most of those things translate quite easily over into what makes writing test suites could come into this code like anything else so we have all these principles that we think help us out when we write software and they're obviously something that you you think about as you're writing software and and they translate pretty well over test as we have these principles like make sure like your objects only do 1 thing or you know make sure that you the try separating things so that things are really simple and composing objects together in this sort of thing but and was those principles translate pretty directly in the testing but to try to keep unit test integration test of any sort of down testing 1 specific thing trying to make sure that you're test are both simple and also transparent because you're not testing them and all these principles that we have for a regular software design apply pretty well the testing and going down slightly to specific we have this three-step process that drilled typically into our heads when writing tests which is that there's 3 steps
to test the set up some stuff then you do some exercise you know do whatever it is that you think you want to estimate animation verification that what you expect that and actually is what ended up happening as applies fairly uniformly across all the different types of tests in end up writing but as a three-step process that like if you actually think through it as a writing tests your test and the clear end up being more self-documenting all the sorts of nice things that we like that test and 1 particular thing that people sometimes say when when we have a three-step process is make sure that you're verification is only 1 assertion by the writer test nature has only 1 assertion and killed Piccu peculiar thing to hear from the 1st time that you hear 1st of all most the time people's 1st thought sir for I actually make that happen from because they remember back to times when they've written tests and their they had this long list of assertions actually make it happen i but even more than that is like when when you hear the same areas and tell you this your resources like what's the actual benefit what am I going for by by King my test down to a single session and as we have used this example here I was fairly simple function it just takes a bunch of dictionaries and enhance them together and and all the keys and associated values together then you have this alternative is green it's better so what what's better about like what what is the actual difference between the 2 examples here why is this any better and causes a simplified example but if the 1st representation of this idea to like make sure you keep your test number 1 assertions and the most obvious benefit just and so that tradeoff is that the most important thing what about the sort of the failures because the server destined to fail but meant fast so 1st year the value of any difference between those 2 slides is basically how much context you get 1 that fails so and the main benefit that we're aiming for with this sort of the idea is that we want more context when test cell and rather than seeing are stuff like well this isn't this which is what you get when you have a source of the like that if you make the sort of larger assertions of then you you you get extra context and that's useful for a lot of reasons 1 of which is that while the 1st 1 tells you that what you got is not we expected that the 2nd 1 tells you not only what you got is and what you expected but possibly the extra information that you've got telling you in what way the actual implementation in your code differs from what you expect it to be so for example you're swapping values for some reason for a protein is obviously in this example it's pretty unlikely from an implementation point of view but in real-world examples it's it's quite common for things like that to happen and if all you see is just well this isn't this but it gives you sort of less information less ability to just other look at and say 0 well that that doesn't look right in multiple places and the contamination of the places where it doesn't look right means that what I did wrong is something and in particular applies unit test to where which has this type of quality protocol where you can get all sorts of nice context like that the so moving out of it so now we sort of like the idea of having 1 assertion in test for the for the reason of of getting this extra context but but sometimes in fact turns out to not be possible oftentimes now we're shifting over from unit testing integration testing and sometimes what happens is that sort of these 2 to sort of worlds of of assertions that you wanna make and sometimes you wanna make a sort but equal lost some assertion of that sort which are basically data comparison assertions there like I have some stuff some values and some expected values on which of those 2 values match up with each other and a lot of times when you're writing applications when you actually want certain at some point when you write your test which is more like I want to say that some collection of things are true that the state of my object application something is true and you just won't have assertions for those because it has know about those they're part of your application but it's it's about basically the difference between making assertions about some data and maybe assertions about the meaning of some application specific things and so to take another specific example from joint some strings call unit testing helped you you just make inequality assertions anything application those things actually HTML unit that's probably isn't gonna help you because while the standard library has much of passes for things it doesn't have assertions for that so if you wanna make assertions about some of 2 pieces of HTML being equal as HTML not necessarily a string literals are you sort of out of luck and that's kind of unfortunate because that's kind of useful to do about but that 1 minute but in test that we write down the this is what we're doing a lot of time and sort of things like changes in weight space and things like that are just annoying so you just comparison HTML that's way more useful makes successfully was real but output is not something that you can find out a lot so here's a pretty specific example that do I have sort of fake but very much
similar to a real example of a test for a enhanced over some web application that basically you get an idea it shows you the associated piece of media with that idea and we have a 3 steps here we do a little set up the summer hand-waving here that basically there's some in-memory database so adding this advertising to and immediate the URL in our application it's actually supposed to be showing that and we make these assertions about OK I expect these 3 things that happen and I wanna make sure that I got the right status code I wanna make sure that and actually properly setting headers for constant type or whatever and then I want to make some decisions about the the body of my response instead this for a moment try apply the rule that you had before which is about 1 assertion protest and it's sort of not necessarily obvious what the way to make that happen is in this case because all 3 of these assertions are useful and they're all things they're basically does my response actually work correctly and and so on the same mind-set that we had before basically meet some people to this but the up splits of the 3 tests with the sort of thing this sort of same set up an exercise non ideal for reasons that I'll skip over at the moment but instead just skip straight to something like this out where I had nothing highlighted because this is way more slightly more code on a slide and I expect anyone to read but so what is this this is basically the conversion of this 3 these 3 assertions in the 1 assertion actually encapsulate some meaning and the meaning of this assertion is actually trying to encapsulate is that responses have content I sometimes 1 certain against that content and there are a couple of things that need to be true when I'm making an assertion and I want this assertion this handled taking care of all those things and so for example we're taking all the same things you're along with the addition of content line so we're taking all these things and making all the decisions in the any assertion method and at the end up with something that looks more like this and obviously would cut down the number of lines in the test I
think also we've sort of gotten some extra clarity I think that the solution with this name has more of a direct it's tell you directly what what you're trying to start against and we also have the same benefit that we get any time that we take a bunch of code and refracted into 1 place which is any time you wanna come back and make some improvements to this slide that's going to immediately go and affect all the test that you have in some positive way and yet you actually obviously have to be careful with that because you can celebrate tests in that way but you are careful with that of what it means is that things like that if someone comes in and makes a change to a test and it starts failing in this assertion and they noticed that it fails in a particular order so for example it'll fail 1st for the content with I may decide that you know what it doesn't it doesn't really that is really as helpful as if I we ordered these assertions if if an assertion fails potentially they wanna see the body 1st and then have that paedophilia message if you got a different response than you expected because that that will that'll will tell you more information on the going here and reverse that assertion and now that that particular thing which is probably true across a test suite you now get their failure messages everywhere and just because someone noticed and when went there and I needed improvement so it's the it's the same benefit that we get on any can we take some software we we take some In particular the operation that we're trying to perform and refactor factored out into 1 place that we can basically concentrated on and and so what happens so what happens is but if you do something like this we are the proposal of starting to build assertions on top of the the comparison assertions is the end up with a sort of hierarchy of assertions so at the bottom you have your data comparison assertions because at the end the day you are just comparing 2 objects so that has happened somewhere but on top of that you can add these layers of meaning and so rather than just being about comparing strings now you have an assertion built on top of that but that's really about comparing HTTP responses but you know at the end of the day comparing a bunch of values but it becomes a much more powerful assertion that's able to add all the nice messages and things like that they might possibly want do not want you know that you're actually dealing with comparing HTTP responses on and then on top of that you can even build more interesting things on some of these things in our case our only in theory they don't actually exist yet much as I would like that they would but but if you come back inside that you know what I wanna do even more fancy things or even more useful things about universe perspective but you can come back and layer on top of these assertions that are making decisions about the responses and directly assertions about was this validates the HTML or layer on top of that something like now I was doing assertions about how these things rendering the browser given whatever conditions I so feel like it placing around and so on were actually doing well on time so I actually spend a bit more on this slide which is and so where we go from here assuming that we have all agree that and this sort of using assertions to build more meaning on top of that data comparison assertions of so what comes out of that and so it's it's kind of interesting what comes out of that I think after a while doing this I what comes out of that is lovingly makes in hell I think by which I mean we build up all these assertions in a bunch of Nixon's in our case and and that's great it gets you kind of benefits and I'll old specifically tell you what what some of these things have before actually tell you why but there is a nice way of doing all this and so for example we do or GBM that's it's like canoe database in in-memory key-value store thing that's in standard library but for example GBM doesn't have ways of is that it's no there's no like object layer on top of that there's no way of basically comparing GDP databases that their files on ceramics and that like will take a GBM database about dictionary interleave those 2 things are equal we have mixing for logging this is something that I think everyone has written at least once there's some packages that actually try to make this helpful for you and it basically attaches a handler to the root blogger for the standard library lying module and then let you make assertions about things that have been logged think everybody has written that I think that assertion is quite often written and rewritten by people we have a mix in there as assertions about our own proprietary log format which is quite crazy so you know percentages something more sane and are able to make assertions against that we have this response content makes in which I mentioned which has this content assertions something as content doesn't has come to have content has constant but what a certain way the stats the with the data dog from stats these like that and then you send the metrics and it basically but lined up eventually graphs and things like that once you make assertions about things having been metric having been incremented all those sorts of things and so what happens is basically you end of growing this after a companion To test suite which is on I have all things on a test and I have all these useful assertions that have come either some exist yet or have some meaning specific to my application and then unable to basically use those in all places in my application but I called that Nixon help because it sort of annoying you use just the the the coupling of inheritance actually adding these assertions you test is kind of annoying so I will mention that there is something else it's up on the slide which encapsulates this idea of I have this collection of things that I that I call insertion of and I wanna just be able to use that all over the place and the test of matches the this sort of thing to solve this problem and so there was checking out of if you're not convinced and and so the last thing that I wanna say it's sort of a call to action which is that In part to myself which is that having a lot of people are writing these sorts of assertions there are things that are useful right like if you wanna comparison HTML where are you going to go and look for the assertion that actually does that In a way similar to how I described sigh I I think we need to start sharing these things that were writing when was the last time that you download and install the package that whose job it was to add a bunch of assertions about I think it's the more common to do that for a test tools matches and the possibly awesome packages that do that but I'm not sure how widespread there's some sure someone will tell me but regardless I think there are a whole bunch there is there's this whole layer of things that that is possible to build on top of the simpler assertions that we have some producers of comparisons and would be nice if we as sort of built up a bit and sharing these assertions of the thing we have to be written once begin can sort of distributed benefit crossed everyone that's what I got the semantic thank you Julian and so on
we have at least 5 minutes for questions so that is 1 microphone the 1 on the right not aggressive in all but the question is not when the microphone to was placed user can 1 question but I would like to know how strict supply you with this wonderful people on stage because if you go back to your 1st example what if your data structure say you have a dictionary is more complex
than that but in this test this is if you only want to
test for cats and dogs if you go into a direct comparison will get bigger this work with with which is not telling you anything right so as you would need to assert statements of look into that so what would you yet so sorry the mean if they are assigned this talk about last week in London and 40 minutes not 20 which is laughable because and someone asked the same question if he is actually the sum so so the answer is or is there is there is no general answer that we were all going the world and just like cave at some point sometimes that the the the ideal answer I think is usually that I try to actually use those cases as examples for 1 to try and look at the code again and see if there is a possible way to split whatever it is that's doing that apart because again like if you have this thing that outputted and there's like there's a bunch of us assertion that you making on pieces of then possibly that means that when you actually really have encode land is this thing that really should be spitting out a whole bunch of different results and then something later that combining so but I try to be fairly straight because I think that's turned out well in that particular case sometimes tells me useful things about the actual code that I'm writing and tells me how to break it up differently but but sometimes it happens thank you and so if
I catch you correctly it's like uh makes less assertions that make them more meaningful by using custom assertions and then that works well I guess you have to generate things that you test for a HTML for instance output and you can never be and share them and then use them but I guess it kind not cases those will be specifically application and I've had something recently with the high test fixture and then I realized she's an putting so much uh logic into this assertion introspection and unit test for this so the kind of writing a test for my fixture and an the afternoon was overnight every got and I kind of went along without thinking is is that really the way forward and making my tests more smarter smarter and certainly better readable but now I have to test at the end of the test assertions and you see any way out of that what's what's your yeah experience it's it's the same thing and the same thing with anything interesting thing which is like it's always a balance of so and that's obviously not good I I don't wanna be ranges from a test and think anybody does but the nice thing I think is that often the assertions that we assertions that we have that we've written come out of life I have a bunch of places where I I wanna make the same set of assertions and I noticed that and I say OK that means that that's a good place to factor factor out on we we try not to start from the other side which is which is why I have this say this thing that I wanna sort without start writing assertion method and then using it in places because then it often does turn out like that the you try shopping a lot of things in that position at the end of day like that the flags in the assertion to do different things so we tried to overhead actually get merged I think there's a flag test we now where the site embarrassing in the video they have that that there is a is a flag somewhere in our tests we prefer doing comparisons on like HTML and it's sort of switches on whether or not to do in a certain order searchable and I hate it so much fun for half year so it happens so unfortunately and I don't have any other like and I don't have any any anything that can help out without other than just use use your best judgment as it's happening the any other questions hold behind you have a question that but which is you got sometimes I could a question that are now so we both work on twisted projects and I know that I've been told of on the times having multiple assertions and but I wonder when you just moved only recessions into a custom assertion right you come out there you can what's JP will always preached to his that's the problem is that if the test is that you have to run the test multiple times to get to find out the the fails on multiple search you have to run the test multiple times to get all the face so what is the word instead of having these multiple sessions in the in the rapid just collecting all the information and putting in a long a single man containing yeah yeah but you can get you can around the sort of questions yeah so so I agree and with him and with that and that sometimes it's nice additional into a container but I think it's easy very easy to be lazy in that case because that is sort of a non-obvious thing and until someone tells you to do that I think you'll like it big people probably think of that when they when you try to apply this rule I don't have to show that in the a couple like be topic comparisons as in all that seems ridiculous and I think that's a solution people like we are likely to come up with some muscle models and some so to be perfectly honest I think that the actual solution for that problem which I understand that but that has tools has this nice the other thing which is of failures that don't actually stop execution of a test point that you wish to explore how much you see an action yeah they're pretty pretty great but so it so if you if you actually have a bunch assertions and you want also get this thing of harmonics skill 3 reason and I want all the context and I think that's the actual where they go as the together get something like that OK great any final questions but there is 1 more time yeah I think so did you have to look into Python because I think it's also some most of the problems to the we describe yet and I'll be perfectly honest I don't know I don't know all the surrounding things that quite a sad so I believe you ensure that it has like symbolist like other than just the eliminating test thing and it was like fixtures and all the other components of desire I mean you don't need to make sure that we because I just shows much more of the context yeah yeah I know that it it shows nice things like but also locals and frames right when when test film thing like things like that I know it has some nice things I don't know which particular like this I can I can imagine it can provide the same sorts of like it's not going to give you later sessions for those are things that you're probably defining application but I know that this will give you some nice things but I know and use them on their OK thank students grades


  331 ms - page object


AV-Portal 3.20.2 (36f6df173ce4850b467c9cb7af359cf1cdaed247)