Django Testing on Steroid: pytest + Hypothesis
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 130 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/50105 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Statistical hypothesis testingHypothesisStatistical hypothesis testingEmailSoftware developerPresentation of a groupSoftware frameworkWebsiteTrailWritingBuildingPrototypeData modelView (database)HypothesisGoodness of fitStatistical hypothesis testingBitPresentation of a groupGraph coloringLimit (category theory)Multiplication signNumberTrailDomain nameoutputLatent heatError messageWritingString (computer science)Endliche ModelltheorieExtreme programmingElectric generatorPoint (geometry)LengthView (database)Client (computing)Plug-in (computing)CodeFunction (mathematics)Projective planeNormal (geometry)Library (computing)Functional (mathematics)Scripting languageUniform resource locatorRepresentational state transferDatabaseContext awarenessSocial classBoolean algebraSinc functionProgrammer (hardware)Software developerWebsiteEmailParameter (computer programming)Dependent and independent variablesCovering spaceFront and back endsComputer configurationMaxima and minimaBuildingQuicksortMeeting/InterviewComputer animation
08:05
View (database)Statistical hypothesis testingType theoryHypothesisUnicodeQuicksortChaos (cosmogony)outputData typeParameter (computer programming)Key (cryptography)Electronic mailing listTupleLengthBitNumberAbstractionMaxima and minimaField (computer science)HypothesisSemiconductor memoryStatistical hypothesis testingParametrische ErregungGreatest elementData structurePhysical systemDependent and independent variablesCASE <Informatik>Series (mathematics)Error messageServer (computing)Crash (computing)Exception handlingString (computer science)Single-precision floating-point formatWeb 2.0Type theoryBoolean algebraMultiplication signASCIILibrary (computing)Point (geometry)Validity (statistics)Strategy gameRange (statistics)UnicodePiCodeGraph coloringComplex (psychology)outputDatabasePreprocessorGoodness of fitExpected valueComputer animation
15:52
Presentation of a groupRight angleMultiplication signMeeting/Interview
16:29
HypothesisData typeoutputType theoryMusical ensembleMultiplicationDifferent (Kate Ryan album)HypothesisParameter (computer programming)Functional (mathematics)Default (computer science)NumberMeeting/InterviewComputer animation
17:36
Statistical hypothesis testingCodeHypothesisBitFunctional (mathematics)Local ringQuicksortLibrary (computing)Product (business)Standard deviationPiComputer programmingMultiplication signMaxima and minimaLine (geometry)outputParameter (computer programming)Musical ensembleExecution unitAdditionSoftware frameworkPlug-in (computing)NumberCASE <Informatik>Unit testingDisk read-and-write headSet (mathematics)Internet service providerStatisticsEntire functionType theoryPhysical systemFunction (mathematics)Control flowOnline chatCodeMeeting/Interview
22:17
Physical systemValidity (statistics)Point (geometry)Cartesian coordinate systemoutputEndliche ModelltheorieElectric generatorStatistical hypothesis testingUnicodeCASE <Informatik>Set (mathematics)View (database)HypothesisCore dumpBlack boxObject (grammar)Strategy gameLink (knot theory)MultiplicationGraph coloringParameter (computer programming)Goodness of fitDefault (computer science)ASCIIMereologyCodeTouchscreenField (computer science)Multiplication signNormal (geometry)Error messageInternet service providerInterrupt <Informatik>Meeting/Interview
Transcript: English(auto-generated)
00:06
So, I think you're ready to go. Ollami is going to be talking about hypothesis with PyTest. So, I think we're interested to talk. So, it's all yours. Good luck. Okay. First of all, thanks everybody for coming.
00:23
This is my first talk ever. So, please be gentle. I'll be going over the Django testing, especially using PyTest and hypothesis for test generation. We're going to see how we can make thousands of tests
00:40
with relatively small code. Now, my name is Boem Milatic. I'm the founder of Terrific. I've been working as Python developer for more than a decade. And I absolutely love talking about Python. So, you can connect with me on LinkedIn
01:02
or on email. Sorry, but I'm not very active on other social networks. So, during this presentation, we're going to cover PyTest, PyTest Django plugin. And after that, we're going to move to using PyTest parameterization to generate the tests.
01:24
After that, we are going to add hypothesis into the mix and see how far we can go. A main point to mention is that we're going to be focusing
01:40
on PyTest and hypothesis, but this is something you can apply to basically everything. Any kind of Python script function, we are talking about in the context of Django, but it's pretty much usable everywhere. Since we only have half an hour,
02:01
we're not going to go much into the dev because PyTest, PyTest Django hypothesis, good practices and stuff like that on its own will take much more time. So, I'm just gonna give you a brief overview of the things and hopefully get you interested to start discovering more things on your own.
02:24
If you have any questions later, feel free to ask me. I'll be on the channel and let's get started. For this example, I decided to make a website for keeping track of unicorns. Basically, we're gonna add new unicorn and list added unicorns.
02:42
We're going to focus basically just one endpoint and I believe we can generate tons of, huge number of tests for that endpoint. Before testing, some of the good practices
03:01
that you know what you're building. Sometimes when you're a programmer, you're going to get a specification and clearly define inputs, outputs and that's perfectly fine. But when you're doing project on your own, take some more time before jumping into the code.
03:20
It can save you quite a lot of time. Also, since in this project, we're going to be working as REST API, you wanna write API specification and then consult your front-end team or some other person.
03:41
When writing API, it's quite helpful to have multiple persons. And once we have API specification set, then you know what we need to do. As you can see, here's the example of the endpoint we're going to be using.
04:01
It uses just two parameters, color and the name for the unicorn. And it returns status cost 201 when it has successfully added unicorn to the database or 400 if we made something up. What we're going to do is make sure to generate
04:21
as much as possible 400 responses. Basically try not to crash the server and get some 500 error. For a unicorn model, it's very, very simple. I try to focus mostly on the testing side,
04:43
which made my Django set a bit weaker. We have one for the colors and then some limitation, pretty simple. So the name must be at least two letters long
05:01
and at max 30 letters long. Now when writing tests, big, big number of tests is going to be located on the domain limitations of a specific value. For example, in the name, if there is some kind of error, it's going to happen either on a character
05:25
that is two characters, the text that is two characters long, one character long, 30 characters long and 31 characters long. Also we're going to test the empty strings. Those are some extremes
05:41
and they cover 99% of the errors that happen. It's quite unlikely that we are going to have some sort of error with a string that is length of 10. So always focus on the extremes. And now we can start with our simple test.
06:03
In this first test, we're sending just empty data. For that to expect our backend API to return us a simple error and tell us we are not doing fine, error 400, invalid request. Ideally, when you're returning requests,
06:23
you wanna provide some text in the body of the request. Basically tell your frontend friends what they need to do to fix this. But since we are doing as minimalistically as possible for this, we're just returning status code, nothing more.
06:44
And the next one is basically checking if the client is, if the unicorn is successfully added. Now, one thing I will mention, you can see that we have some nice fixtures in PyTest. That's the client and client DB.
07:03
Now that's all thanks to PyTest Django plugin. Client will allow us to basically use Django as you would use the normal request library or something like that. As you can see, I'm not calling the view function directly,
07:22
but using the URL, which is quite handy. And the database fixture in the second test is basically telling us that this test will need access to database, otherwise it's going to crash. So those are the two fixtures
07:42
that you're probably going to use the most. Onto the next things. And here's our simple view. Basically, I created the class base view. Since in normal situation, we're not going to have just post.
08:01
We're going to have get, delete options and a bunch of other stuff. But for now, just post. If you can see my validated validation here is basically just checking if dictionary is empty. If yes, return HTTP 400 bad request,
08:26
which is perfectly fine because my test here is passing. And afterwards, I'm just saving it and it's all good. When doing TDD, it's very important to not get ahead of yourself.
08:41
I love writing tests that fail. Because when you write a test that passes right away, you don't know if your test has any errors in it. If you're testing with wrong or if everything's perfectly fine. It's much easier to debug when your test crashes
09:00
then you fix the code. Okay. And now we're going to use Pytest Paramount Trace. It works quite simple. Basically, just type this, give it a name of the parameter, which is going to be Paramount Trace. Here it is key and the list of values.
09:24
Now, as you can see in this example here, it's very, very basic. But the idea is we don't want to repeat the code. We could easily split this into two tests. One that has only name in the request and one that has only color in the request.
09:41
Like this, we only have one test. It's quite trivial in this example, but when you have a much more complex request structure, it's going to save you quite a lot of time. Almost regarding the Pytest Mark Paramount Trace, you can supply multiple parameters that you wish.
10:04
Behind the key in the, as a first parameter, you can just add comma and list the second parameter. And instead of the list of values, it's going to be a list of tuples that are the same length as the number of parameters. Okay.
10:23
So next thing we're gonna do is once we edit test that validate structure of the request is basically try to send some junk data to our end point.
10:41
Because also a good tip when testing is that majority of your tests are going to be located at the borders of your system. That means you're going to have to extensively test your API where you are receiving requests from the user and the third party libraries that you're using
11:01
where you are sending requests and receiving responses. Since in majority of the cases, we're going to get the wild stuff from our front end team, which would be using this API. But they may get something wrong or some kind of malicious user is going to send us
11:21
some crazy stuff that we're not expecting. For that, we need to try throwing everything at the API and hopefully getting just error 400, which is user request is invalid. What we don't want to do is get error 500,
11:40
which is a series of errors that correspond to server crashes and stuff like that. Basically, we want to be able to graceful handle every exception possible. Now, as you can see, we got from here, three tests above from this, we got two tests.
12:04
So that's five tests and we just wrote two. And it's time for hypothesis. Now, hypothesis is a property-based testing library. It uses Haskell as a starting point
12:26
and it did quite a lot of good stuff that we can use. Basically in hypothesis, you specify your strategy. Here we are saying the name is going to be a text and the text is going to be minimum size two
12:44
and max size 30. The same thing we got when we define the field for the name. Now, usually what's gonna happen is we could write, for example, in Pytest parameterize,
13:02
just pass the range of values as the string length from two to 30 and basically multiply character A or B by that number and get the string length, which is good and which is going to work.
13:21
But like almost every single web input requires text. We're gonna get Unicode. And hypothesis is gonna provide us with bunch of crazy stuff ASCII text, Unicode text, characters,
13:41
invalidly formatted ASCII inputs. And what we want to make sure is that we get a status code 201, that everything is set in the database as it should be. If you're doing some pre-processing, lots of time this is where we're gonna catch
14:01
Unicode errors with the text. Okay, now that we have successfully tested this case, we wanna test all the invalid cases as well. For example, we can set up one strategy for text
14:23
that is using maximal size of string one. It basically went to zero and one. It's not as quite complex in this case, but I'm just asking for a little bit of abstraction
14:41
because you can apply this to pretty much anything. Also, okay. Next thing, we are also going to test the data outside of maximum size.
15:01
That will allow us to test also a bunch of crazy stuff. Now, the reason we're limiting the size, we're not putting 10,000 or some other stuff is because in SQL when you define a field, it has a fixed size in memory.
15:26
Okay, with this, we did some basic overview of hypothesis by this and by this parametrize. Now, I'm open to the questions.
15:40
Feel free to ask me anything and that way I can go more into depth regarding concepts that interest you. So for anyone who wanted to ask a question, there is a Q&A bottom. They need to click there and ask a question. It's also possible if you click in the right hand and it can allow you for the question.
16:02
So if anyone wants to ask, now is the time. So thank you very much. Okay. Okay, any questions?
16:23
My presentation was so good that nobody had any additional questions. You were, seems that you were really, really good. We have a couple of questions actually. We have two now, we have two now. So one person is asking, what is the ad given decorator?
16:43
Oh, yes, I can explain that. That's a very good question. Given decorator is provided by hypothesis. You're basically telling here, like we did here with by parametrize, we're telling this function,
17:01
we're gonna supply multiple values for the name. And here we are saying, okay, hypothesis is going to supply you multiple values for the name and the generator. By default, this is going to supply you 100 different kinds of texts
17:21
that correspond to these parameters. There is, you can use another decorator about this to set it to a higher or lower number. So this is the hypothesis decorator. So the other question is,
17:40
what's your feeling, what's your thought about pytest versus pyunit? If you have any opinion on that. Well, it's a difficult question for me because when I started, I was using a unit testing and it was pretty much similar
18:02
like in other frameworks for me. But when I started using pytest, it felt more platonic to me. Tests are much prettier and I like the fixtures and bunch of other stuff that gets provided here. So I prefer using pytest
18:25
mostly because it's my kind of the style of programming. I don't think you can get wrong with unit tests as well. An important thing is to have tests, find the framework that works best for you.
18:42
One plus side for pytest is that it has plugins for pretty much anything. I'm not sure what's the situation with unit tests. So next question is, how do you define the number of cases that hypothesis has to generate?
19:03
Oh yeah. Above given, you have another decorator, which I don't know at the top of my head. I'll type into the chat after this. You can set, it's at settings, I believe, and you define number of how much you want.
19:24
When running pytest, you can provide additional argument. Dash dash, show hypothesis, this statistics and it will show you how many tests it has generated for you.
19:44
So please write me in the chat channel and I'll send you the code examples and we can discuss further if you like. I'm just going to repeat that. There is a talk, Django testing, and I'm sure that that's the channel that you can follow the discussion.
20:01
So one more question. Robert is asking, do you have any recommendations for applying tests to an existing code base? Yes, now this is something that happens quite often. You get the code base that has absolutely zero tests.
20:26
First of all, before you even touch the code, you are not going to start with tests like this. You're going to have to start with functional testing. Basically start large, test entire system and outputs
20:44
to make sure you don't break anything, large functionality. And once you get that down, then you can start moving code around. Then you modify a little bit of code, write the test for that, see if anything breaks down.
21:03
Don't start with the functions, start big. That's the way I did it and it usually works. Because that way you discover, oh, in my local setup, I'm missing this sort of library. They did something crazy in the production.
21:22
I also need that. So first start with the large functional tests. I would even go so far to recommend Selenium for that. And then get smaller and smaller. It's when you usually get a code base
21:42
that doesn't have any test. Coding standards are not quite high. So you might get a function that has, I don't know, one tons to tons of code. The most that I saw was 5,000 lines of code and 45 input parameters.
22:00
Stuff like that, you cannot start to break apart until you have dozens of tests that test existing behavior. And only then you can start slowly breaking down into smaller pieces. Okay? Cool, cool, thank you.
22:21
So I have one question more from Discord. Javier is asking, in case you need to test with more complex inputs, like a self-made object, how will Hypothesis handle this? If it's a Django model, Hypothesis has a very nice support for that.
22:44
You basically just tell it to use this model and it will generate a random values for that model and supply it. You can also make your own strategies for generation, link multiple strategies together.
23:00
For example, if I wanted to add more unicorns here, I would use, I could use a name and color as input parameters for Hypothesis. Okay, good. So we have more questions, so I go to continue
23:21
because we have a few minutes. Jennifer is in, okay, seems that she missed some part because it was a three-year-old person interrupting. So who did you generate the 400 tests? Here, well, when you say given,
23:45
that decorator basically tells, okay, I'm gonna turn this, provide, I'm gonna use this strategy to generate various values for the name. And by default, it's 100 values.
24:02
So this piece of the code that is currently on the screen, that is 100 test. Yeah, you can set it up above and it will generate 400 various tests. Now the text field, which I really like
24:21
in the strategies for Hypothesis, will try to break your code in ways you didn't think it was possible because it will throw Unicode, ASCII, characters that are invisible, stuff like that. Also, invalidly formatted Unicode.
24:41
Basically, all the crazy stuff you can expect from your users. I hope that answers it. I think it does. Okay, last one, last one, because we already did like 10. Daniel is asking, he always has the impression
25:00
that testing something a lot of times doesn't do much more than that few careful thought like with the tailored normal test. And he's asking if you have an example when Hypothesis was catching an error. So when Hypothesis was caught, it's cases like because you were doing
25:21
like a really big set of tests. So I think that's the question. Sorry, it was like maybe changing it. Yeah, basically, I kind of agree with his approach when you're doing testing inside your own system
25:40
because then you are already, my philosophy is this, you have user input, you have one giant wall, you're gonna do lots of data validation and stuff like that. And that's going to be, most of your tests are going to be concentrated. Inside your system, you can trust it
26:01
to provide valid inputs and stuff like that. So there you can use carefully design tests and not treat it as a black box. If you can see here, I'm never calling the unicorn view. I'm treating it completely as a black box.
26:21
So from my standpoint of view, if I didn't use Pytest Django, this could be a flask application or something in Java, anything at all. Does that answer the question? Yeah, I think it does, I think it does. Okay, thank you very much.
26:41
That was a really nice talk. So I need to play some because, whoa! Thank you. Thank you.