Keynote - Testing in Django
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 14 | |
| Number of Parts | 48 | |
| Author | ||
| Contributors | ||
| License | CC Attribution - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/33195 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
DjangoCon US 201714 / 48
2
5
6
14
16
23
26
30
31
32
34
39
43
48
00:00
Statistical hypothesis testingFood energyStatistical hypothesis testingXML
00:45
TwitterSelf-organizationVideo gameState of matterArithmetic meanIdentity managementIntegrated development environmentTouch typingTerm (mathematics)Software engineeringRoundness (object)Software developerXML
02:16
Level (video gaming)BitRight angleMultiplication signXMLUML
03:06
Execution unitSoftware frameworkStatistical hypothesis testingIntegrated development environmentStatistical hypothesis testingEndliche ModelltheorieQuicksortIntegrated development environmentResultantUnit testingMobile appBitMultiplication signSocial classPoint (geometry)Parameter (computer programming)Data managementClient (computing)Software frameworkCartesian coordinate systemTask (computing)Revision controlFunctional (mathematics)Term (mathematics)Beta functionVariable (mathematics)Web browserComputer fileLevel (video gaming)Message passingConfiguration spaceObject-relational mappingCoefficient of determinationComputer animation
04:58
Client (computing)Template (C++)CASE <Informatik>Web browserView (database)Interface (computing)Client (computing)Dependent and independent variablesState of matterConstructor (object-oriented programming)Unit testingStatistical hypothesis testingCASE <Informatik>Template (C++)Task (computing)Extension (kinesiology)Revision control1 (number)GenderInverse elementLibrary (computing)Structural loadComputer animation
05:56
Computer configurationClient (computing)Database transactionRollback (data management)DatabaseInteractive televisionSet (mathematics)Revision controlGenderClient (computing)Statistical hypothesis testingRepresentational state transferTablet computerDatabase transactionBlock (periodic table)Rollback (data management)CASE <Informatik>Task (computing)NumberDatabaseSoftware frameworkFilter <Stochastik>Software bugState of matterError messageDifferent (Kate Ryan album)Insertion lossSocial classDefault (computer science)Point (geometry)BitPhase transitionField (computer science)Functional (mathematics)Interface (computing)Suite (music)Ferry CorstenEvoluteComputer animationXML
08:51
Interactive televisionDatabaseMathematicsStrategy gameDefault (computer science)DatabaseMultiplicationStatistical hypothesis testingPhase transitionReplication (computing)Integrated development environmentQuicksortParameter (computer programming)Task (computing)Unit testingRevision controlResultantImplementationRight angleSet (mathematics)TriangleWebsiteSoftware frameworkData dictionaryConnected spaceXMLComputer animation
10:39
CASE <Informatik>Statistical hypothesis testingDatabase transactionServer (computing)DatabaseRollback (data management)Client (computing)Social classDatabase transactionStatistical hypothesis testingUniform resource locatorRevision controlCASE <Informatik>Default (computer science)Attribute grammarSoftware frameworkClient (computing)Extension (kinesiology)Dependent and independent variablesFamilyServer (computing)DatabaseNumbering schemeObject (grammar)Projective planeCoefficient of determinationRight angleState of matterHeegaard splittingSet (mathematics)Latent heatLevel (video gaming)MereologyQuery languageSequenceNumberGroup actionMultiplication signElectronic mailing listOrder (biology)BitFactory (trading post)Software developerCommunications protocolStructural loadService (economics)Point (geometry)Basis <Mathematik>Equaliser (mathematics)Patch (Unix)HTTP cookieConcentricComputer animation
14:15
Statistical hypothesis testingStatistical hypothesis testingFile systemMessage passingRevision controlComputer fileDirectory serviceLine (geometry)Mobile appReal numberLatent heatEndliche ModelltheorieTerm (mathematics)Pattern languageComputer animation
14:56
Server (computing)Unit testingIntegrated development environmentFluid staticsCASE <Informatik>Revision controlComputer fileStatistical hypothesis testingMultiplication signPoint (geometry)Real numberClosed setRootClient (computing)Video gameRegular graphExecution unitComputer animation
16:08
Client (computing)Atomic numberTexture mappingCASE <Informatik>Statistical hypothesis testingParallel portCASE <Informatik>Client (computing)Strategy gameStatistical hypothesis testingData managementMultiplication signBlock (periodic table)GodResultantProjective planeError messageTraffic reportingState of matterMathematicsRevision controlType theoryUnit testingParity (mathematics)Process (computing)Different (Kate Ryan album)1 (number)Context awarenessDatabaseModule (mathematics)GenderAreaConnected spaceDatabase transactionComputer configurationStress (mechanics)Substitute goodQuicksortSoftware frameworkNumberPlug-in (computing)MultiplicationSocial classEndliche ModelltheorieMehrprozessorsystemPoint (geometry)Similarity (geometry)WordFerry CorstenSuite (music)Ultraviolet photoelectron spectroscopyMaxima and minimaComputer animation
21:19
Statistical hypothesis testingData managementComputer configurationEmailSuite (music)Statistical hypothesis testingDataflowSoftware frameworkSuite (music)MultiplicationTestbedData managementIntegrated development environmentVolumenvisualisierungInstance (computer science)Point (geometry)Order (biology)Line (geometry)Projective planeComputer configurationEmailZoom lensNumberUnit testingSystem callEvent horizonTotal S.A.Library (computing)Translation (relic)Structural loadError messageSocial classDatabaseCASE <Informatik>State of matterFront and back endsResultantElectronic mailing listAsynchronous Transfer ModeWeightDirection (geometry)Functional (mathematics)Task (computing)Set (mathematics)BuildingFormal languageJSONXMLUMLComputer animation
24:21
Social classStatistical hypothesis testingCASE <Informatik>Query languageDatabaseClient (computing)Database transactionServer (computing)Thread (computing)Statistical hypothesis testingSocial classDifferent (Kate Ryan album)Revision controlHierarchyFluid staticsDatabaseDatabase transactionServer (computing)CASE <Informatik>Client (computing)Utility softwareThread (computing)Computer fileAdditionUnit testingInteractive televisionService (economics)Line (geometry)Query languagePoint cloudInsertion lossVideo gameMultiplication signXMLUMLComputer animation
25:45
Client (computing)Factory (trading post)Dependent and independent variablesStructural loadUtility softwareStatistical hypothesis testingMobile appWeb browserCore dumpFactory (trading post)Dependent and independent variablesTemplate (C++)Group actionHTTP cookieAxiom of choiceServer (computing)Client (computing)Matching (graph theory)Context awarenessResolvent formalismObject (grammar)ChainException handlingSet (mathematics)Event horizonMiddlewareVolumenvisualisierungXMLUMLComputer animation
27:29
Random numberTexture mappingFactory (trading post)HypothesisStatistical hypothesis testingCategory of beingEndliche ModelltheorieIntegerElectronic mailing listCASE <Informatik>CollaborationismComa BerenicesImplementationOperator (mathematics)Insertion lossConditional probabilityLogical constantLogicProcess (computing)Statistical hypothesis testingRandomizationoutputSpherical capCodeLibrary (computing)Point (geometry)Software bugKeyboard shortcutCategory of beingMereologyEndliche ModelltheorieProjective planeFactory (trading post)Exception handlingHypothesisInstance (computer science)Limit (category theory)Electronic mailing listStrategy gameInternet service providerLine (geometry)Operator (mathematics)Functional (mathematics)Condition numberImplementationWeb-DesignerRule of inferenceGroup actionLevel (video gaming)Machine visionDifferent (Kate Ryan album)SequelCASE <Informatik>Multiplication signLogical constantCollaborationismSinc functionGenderStatement (computer science)Revision controlDatabase transactionModule (mathematics)Real numberLogicSource codeComputer programmingSoftware developerQuicksortState of matterAdditionAuthorizationResultantBranch (computer science)Cross-correlationSet (mathematics)TwitterPresentation of a groupFocus (optics)Local ringWordSingle-precision floating-point formatMetreNumberAnalytic continuationMobile appUnit testingUtility softwareTotal S.A.Structural loadAbstract syntax treeJSONXMLComputer animation
36:50
Statistical hypothesis testingHuman migrationData modelSheaf (mathematics)Statistical hypothesis testingMereologyUniverse (mathematics)InternetworkingWeb pageMobile app2 (number)VideoconferencingHypermediaRun time (program lifecycle phase)Uniform resource locatorTask (computing)Multiplication signSoftware frameworkLevel (video gaming)Dependent and independent variablesGame controllerDecimalCodeNumberTotal S.A.Functional (mathematics)Projective planeQuicksortJSONXMLComputer animation
38:32
FeedbackLoop (music)Statistical hypothesis testingObject (grammar)Asynchronous Transfer ModeExecution unitRange (statistics)RobotDialectData modelFactory (trading post)Product (business)Abelian categoryCodeDependent and independent variablesClient (computing)Inheritance (object-oriented programming)Router (computing)DatabaseoutputFunction (mathematics)Instance (computer science)Dean numberFeedbackSocial classStatistical hypothesis testingSoftware developerCycle (graph theory)Software frameworkFactory (trading post)Router (computing)ResultantCalculationUnit testingEndliche ModelltheorieoutputInstance (computer science)GenderTask (computing)CASE <Informatik>Multiplication signFunction (mathematics)Parameter (computer programming)FlagProduct (business)Flow separationObject (grammar)DatabaseNumberInteractive televisionINTEGRALPoint (geometry)BitRevision controlFunctional (mathematics)Form (programming)Greatest elementExecution unitConstructor (object-oriented programming)Design by contractAsynchronous Transfer ModeCore dumpWeb 2.0Different (Kate Ryan album)PasswordKey (cryptography)Set (mathematics)Decision theorySemiconductor memoryCodeConnectivity (graph theory)CountingProjective planeCategory of beingRoutingMereologyView (database)Universal product codeNetwork topologyProgramming languageBit rateProxy serverBranch (computer science)1 (number)Mobile appUltraviolet photoelectron spectroscopyPlanningFunktionalintegralAttribute grammarMixed realitySuite (music)Similarity (geometry)Information securityLoop (music)XMLComputer animation
47:29
Mobile appCalculationForm (programming)Software developerDifferent (Kate Ryan album)LogicStability theoryView (database)Extension (kinesiology)JSONXMLUML
48:12
Statistical hypothesis testingTerm (mathematics)LogicFunktionalintegralUnit testingStatistical hypothesis testingObject (grammar)GenderForm (programming)Group actionBuildingDatabaseData managementTheory of relativityComplex (psychology)Connectivity (graph theory)INTEGRALView (database)Mobile appBit rateComputer animation
49:00
Coma BerenicesXML
Transcript: English(auto-generated)
00:23
Oh, that's just repeating myself. I'm full of energy to get started with a day. Yes, we are. And we're going to get started with a very long talk, a rough overview of testing
00:42
in Django and within Django. I would like to say thank you to the organisers and the volunteers who make DjangoCon US happen. Because you and I, we have no idea how much effort it takes to bring a conference to life. And not only that, but also create
01:01
a welcoming environment where we all feel safe. So let's just give them a round of applause. You might be wondering, who am I? And this is my Twitter handle if you want
01:22
to get in touch with me. There's nothing you probably haven't heard about me because there's nothing outstanding that I did in my life. Generally, I'm the kind of person who goes to work every day and tries to be the best developer I can become. I usually
01:42
say I'm in a long-term successful and happy relationship with Python. I've been doing Django for the past couple of years. And in terms of community involvement, I usually do little things. Currently, I'm co-organising the London PyLadies meet-up, and I helped
02:06
organise DjangoCon Europe last year. I sometimes mentor people, and most often I give talks at conferences. I recently joined a company called Street Team as a senior software engineer.
02:22
So yes, questions. I won't be taking questions on stage, and there are two reasons for that. First is we actually might run out of time. And the second one is it never works out for me. I will be honest. While I'm on stage, I'm a bit nervous, and by the
02:41
end of the talk, it's getting a little bit tricky to think straight and give a proper answer to a question, and I feel like I'm actually doing a disservice. So let's take it off stage. I will be at DjangoCon for the rest of the conference until Friday, so come and come to me if you have questions, comments, or feedback. I will be happy
03:03
to have a chat with you. Right, so now that we got to know each other a little bit better, let's get started with testing. We're going to take a trip all the way from version 1.0 to 1.11 and see how the Django test framework evolved over time. It all started about 10,
03:26
11 years ago, sometime before getting Django out of beta. This ticket gets created. Add unit test framework for end-user Django applications. And it lists all sorts of benefits that you can have while having an integrated framework to test your apps. But there's this
03:47
killer argument. As an added incentive, this is a feature that is present in Rails. And that's how it started. It started with a management command, test. It accepts
04:03
pseudo pass at the time that point to test. So you do app.testclass.testmethod. We get a test runner. A test runner is also a config variable that points to a function that runs your tests. That means that from the very beginning, anyone can write their own test runner and test discoverer. The test runner setups the test environment, and it looks up
04:25
for tests in files test.py and models.py. At the very end, the test runner teardowns the test environment and returns the results. At this point, you might be squinting and thinking, models.py? Is she serious? Well, 10 years ago, apparently, dog tests were
04:44
very popular. And they were used to test ORM a lot. So models.py were the place where model and application level tests were meant to be. And everything else was left to test.py. Client makes its appearance as a basic stateful browser mimicking stop
05:05
that constructs requests, passes them to the views, and returns a response, bypassing all networking altogether and interacting directly with the WSGI interface. At its core, client, it is the same client that we have today. It's just a more limited one.
05:22
If you're writing your test using Python unit test library, then you've probably heard of test case. Django also has its own version of the test case, which is an extension of the unit test test case, because it can also do extra things like load fixtures and clean the stubby mailbox and initialize the client and clean the data after each
05:47
test. And it provides a couple of assert helper methods, for example, like I said, redirects and template queues, the ones that we actually still have today. Version 1.1 comes shortly after with a revolutionary set of HTTP methods.
06:04
So RESTful APIs are not yet the next big thing, but Django's looking into the future, and as to the client put, had, delete, and options. Along with the test case, we also get a transaction test case. This is where we need to make a clear distinction
06:21
what is the difference between a test case and a transaction test case. So a test case acts like a burger, because it wraps each test into an atomic block. It enters the transaction, it runs the test, the test does messy things to the database, and finally the test case rollbacks this transaction and leaves a clean slate for the next test.
06:42
Because of that, the test case will always prevent any commits or rollbacks to the database, because it wants to ensure that at the very end, when you rollbacks the transaction, the test restores the database to its initial state. And this is a big performance booster to the test case and for us to writing our tests. If you want to use
07:06
transactions, that this is where you need to use transaction test case. It will clean up after each test by flushing the database at the start of each test. And by the way, the test case from the previous version 1.0 was doing exactly that. 1.2, Django
07:24
defines a new interface for test runners, so we're switching from a function to a class. The class should have a method called run test that will act as an entry point. By default, Django provides a Django test suite runner. So this is not really related to the evolution
07:45
of the Django test framework, but it's a silly bug that I really like, so I decided to put it in. So the test runner returns the number of errors plus failures, all summed together, and what Django does, it spits out this number of failed tests as a sys exit. So if you have,
08:05
if no tests have failed, then it will naturally exit with a zero, which is a success. If you have one test failure, then this is going to be one, which is a failure. That looks correct. If you have 42 failed tests, that'll be fine as well. If you have 256, it suddenly becomes a success.
08:27
Whoops. This is because sys exit number is an 8-bit number, so it overflows into a zero when you have 256. And talking from experience, it's not very hard to get 256 failed tests,
08:47
so we fixed it. We just say sys exit one. So now we'll always either get a zero for success or a one for a failure. 256 now is a failure, as it should be. Let's take a look at the
09:03
definition of the test runner because something changed from the previous version. There's a new argument, fail fast. It allows you to terminate the test run on the first failure, and I find it very handy. The implementation has been removed ever since from the Django test
09:22
framework because it's now available within the Python unit test lib, but good to know it exists. Django has support for multiple databases with a replication strategy, so you can have a primary database and multiple replicas can be created. This is how it looks like in db settings.
09:41
There's a default database and there's a replica. If the test mirror setting from the replica dictionary is missing, then there won't be any replication happening during your tests. Just the default database is created and tested against. If you want to test replication,
10:00
then you need to set up a test mirror, and if you want to use it today, settings have changed, so it should look like this. So let's go back to our databases. When the test environment is configured, a test version of the replica is not created. Instead,
10:21
the connection to replica will be redirected to pointer default. So as a result, the rights to default will appear on the replica, but not because there's some sort of data replication happening between the two, but because it's actually the same database. 1.3, we can now do
10:41
a set query set equal and a set number of queries, which is handy for the test case. We also see a split of the two roles that the client embodied. So the role of manufacturing requests is turned over to the request factory, and obtaining the response was left to the client.
11:00
The request factory becomes part of the public documented API. So how do you manufacture requests with the request factory? Very simple. You just create a whiskey request object with a set of whiskey and request specific attributes like the protocol, the request method, payload, cookies, URL scheme. That's pretty much it. Doesn't do anything else. Very simple,
11:26
very factor-ish. So dog tests. In the beginning, it seemed like such a great idea. You get both tests and you get both documentation. In the long run, it became a nuisance to debug.
11:42
It also became apparent that dog tests aren't really good at providing neither high quality documentation nor comprehensive tests. That's why dog tests will be discouraged to be used in the future. Django is a good example of a big project that came through a love stage
12:02
with the dog tests and just proved that this relationship can't work. 1.4. In version 1.4, the family of the test case classes receives an extension pack. So we get the simple test case and the live server test case. Simple test case is very light
12:21
because it doesn't hit the database. Live server test case runs an HTTP server so that you can use a testing framework like Selenium for your end-to-end tests. And it is based on the transaction test case. Version 1.5 is a time when Python 3 support landed in Django.
12:41
2.5. We talked about the transaction test case and we said that it's flushing the database before each test. Well, at the time, other test runners become popular, for example, nodes and developers figured out that there's a bit of a mess left by the transaction test case. So it has been updated so that now we flush the database,
13:04
not before each test, but after. Let me show you. So if you flush the database before the test, you get the following sequence of actions. You flush, you run the transaction test case, and then you can run your test cases. What happens is that right after the running
13:21
the transaction test case, you get some dirty state right in there. So your test cases actually start with a not clean database. That's why Django default test runner is always rearranging tests so that the test cases run first and then the transaction test cases come.
13:42
And even though the transaction test case leaves some dirty state in there, it actually doesn't matter because we're going to destroy the database. But we can solve this problem in a different way. We can just flush after and then all of our problems are solved and it doesn't really matter in what order we're running our tests.
14:04
1.6. So there's one more HTTP method added to the client. This is patch and if you thought this list is complete, it is not. In version 1.6 we see various improvements. We get a new test runner that is able to locate tests not only in test.py and models.py,
14:24
but anywhere from the installed apps as long as it matches specific file patterns. The new test discovery also means that we have no more tests in models.py. Hooray. Pseudopaths are also removed. Now we can use real file system or dotted directory paths.
14:44
This also allows running tests that are not inside a Django app listed in installed apps. And finally, doc tests won't be discovered anymore and we say farewell to them. 1.7. So three versions before 1.7 Django introduced unit test two.
15:03
Unit test two is a back port of the unit test features that work with older versions of Python and because Django at the time was supporting Python 2.4 it was rendering unit test two. 1.7 Django drops support for older versions of Python and so it removes the unit test two
15:21
and we can just use the regular unit test. At first when the live server test case was introduced it was serving all the static files as if you were running your server with debug equals true. From this point on live server test case will serve only files from the static root
15:42
trying to simulate the real production environment as close as possible. If you want to serve all of the static files then you can use the static live server test case. In 1.8 we get the last known HTTP method to the test client. Can you tell me which one
16:00
is that? Which one are we missing? HTTP spec someone? Okay okay this is trace, this is the last one and once this one was defined the client is complete. Test case and its burger strategy was
16:20
a performance booster and around version 1.8 a few other intricate changes were introduced that made the test case even better. Before that think version 1.7 what test case did is wrap each test into a transaction but it did it in the following way you would enter the atomic block
16:40
and you would load the fixtures you would run the test you would exit the atomic in other words roll back and then you would close all of your database connections and you repeat that for every single test that you have. With the arrival of Django 1.8 we can do something better so we actually wrap all of our tests from a test case inside
17:05
of a different atomic block that is going to happen only once and what we are doing we are loading the fixtures and we are closing the connections a single time rather than doing it for every other test. So now every other every test within also runs within its own transaction
17:26
block but we get also the benefit of closing the connections and loading our fixtures a single time rather than doing it every time. 1.9 whoever works or worked on a large project
17:43
knows the pain of waiting for tests to finish from my experience that would take the pain starts with about 20 minutes sometimes it's 40 depends how large it is and most of us know that the quick easy win for that is to parallelize our tests and also
18:03
that's the time when you find out how many non-pure tests you have. Django 1.9 comes with built-in support for parallelizing tests using Python multi-processing module. In fact Django's test suite is also parallelized. How does it work? The test runner spawns a number of workers
18:22
so if you don't specify how many it will pick up the max number of holes that you have. For each process we'll get an instant its own database so each each worker gets its own database therefore if you have four workers you're going to get four databases. Test discovery
18:43
builds a test suite which is partitioned into chunks of test case subclasses. Workers pick up those substitutes of tests and run them once they're finished they pick up the next one and they're finished and they pick up the next one until they're done and they return the result.
19:02
If you're stuck with a version of Django prior to 1.9 then you can use knows multi-process plugin. It's also using Python multi-processing module but the main difference between the Django ones and the knows multi-process is that you get only one database so you get a shared state
19:25
and because of that you sometimes can get bizarre failures and errors. 1.10 My favorite feature that was introduced in 1.10 is tags. You can tag your tests creating buckets
19:42
of certain types of tests that you'd like to run together or you would like to exclude from your test run. Similar feature is implemented by other test runners like knows and pytest. If you're not on Django 1.10 you always have the option to use
20:00
knows a trip plugin or the pytest custom markers. And finally 1.11 we get all sorts of speed ups and improvements in fact with every version we continuously try to get better in different areas of the testing framework.
20:21
But there's one feature that caught my eye which I think is nice to know about. So starting with Python 3.4 unit test module adds a context manager called subtest. It says a subtest executes the enclosed block as a subtest. In this example what happens is that the test won't exit on its first failure when i equals one because one is an odd number.
20:47
It will actually run through all of the options and it will return all of the results for all of the values of the i. So now with Django 1.11 you can do that you can write your
21:04
subtest when running tests in parallel and Django will report all of the results correctly. If you're with pytest you've probably this probably looks familiar to you because pytest has parameterized. We went through the history of the Django test framework and we've touched on
21:24
different topics here and there but let's see the flow of running a test suite as a whole or what happens when you run manage.py test. So let's split up the test bed into multiple pieces and analyze each one of them and see what happens under the hood.
21:43
So in the test environment we work the management command test and if we zoom in came back and if we zoom in there's pretty much a straightforward invocation. So we're getting the test runner and we're creating an instance of it passing a bunch
22:04
of options and we run the tests. Let's zoom in inside run tests. So the first line in run test is setting up the test environment that will do a bunch of things. So the main free key point of this method is setting up the lockmam email backend so that all the email that we're
22:25
sending during tests is stored into a stub list that we can inspect and we can access it easily. This is a very reduced email backend in its functionality. We also get the Django test
22:43
renderer is replaced by an instrumented test renderer. It sends signals to let others know about various events that happened during rendering and finally we deactivate translation so that during tests your project will the test will respect your project set language.
23:04
The next step after setting up the test environment is to build the suite of tests. The heavy load of building the suite is done by the Python unit test library but Django extends it because it allows you to specify tags and number of calls and debug mode and all the different
23:22
things. So it builds up a suite of tests. This is an aggregation of instances of test case classes all collected in a single place and there's nothing more than that. The next step we set up the databases, we run the checks, then we run the test which is by the way completely
23:42
delegated to the unit test text test runner and for the cleanup we tear down the databases. Afterwards we need to tear down the test environment. This is pretty much the reversed order of what we did in the setup test environment so we need to bring back the original email
24:02
backend, the original test renderer and finally we delete the state and the mailbox. Last step we return the results of the test suite by combining the number of total failures and errors and these lines should probably look familiar to you at this point.
24:22
A quick recap on the different test classes and how they relate to each other. So this is the hierarchy and the simple test case is at the very top subclass by the transaction test case which is subclass by the test case and the live server test case, finally specialized version of the live server test case which is the static live server test case.
24:45
I've been saying test so many times. Simple test case is very very fast because it doesn't interact with the database. You can't query, you can't save, you can't do any of those things. It has access to the test client though so think of it as a slightly more advanced version of the
25:05
usual python unit test test case. Transaction test case is not fast at all but it allows database queries and transactions. Test case is faster than the transaction test case but not as fast as the simple test case. It will allow you to do database queries but will restrict
25:23
transactions. Live server test case will act as a transaction test case and it will launch a live http server in a separate thread from the thread that's running the tests. And finally the static live service is the specialization that in addition to
25:41
everything else will serve all static files. Client is a very handy utility when you're testing your Django apps and the three core actors involved in browser mimicking are the request factory, client handler and the client. We've talked about the request factory being the
26:03
most straightforward actor. It constructs requests and it encodes data. Client on the other hand performs more actions. First off the client is stateful so it retains cookies and hands
26:20
sessions for the duration of the test client. It also does a few other things to the response. So we talked about the instrumented test renderer. Well the client is listening to those events. Events like template rendered and request exception
26:41
and that's why it is able to list all of the templates and template contacts that was used to render your response. It also sets a couple of other things on the response for example the original request, the client object, JSON, a resolver match. It can also handle redirects
27:01
and build a redirect chain that you can inspect. Main goal of the client handler is to return a Django HTTP response with the whiskey request attached to it. It will load all the middleware that you've set in your settings. It will disable the CSRF checks which can be enabled
27:21
back if you want to and it tries to emulate as close as possible server and browser behavior. We've talked about all the different tools that Django provides for testing but how do we
27:41
make sure that the tests we write are reliable, future-proof and fast enough? Reasoning about quality is not easy. Besides our intuitive human understanding of what quality is, there are some tools that can help us and let's start with something simple.
28:02
Introduce Factory Boy to your tests. Lots of people like it because it has this shortcut methods for creating models. It generates random yet realistic data because it's based on a different library called Faker and Faker has providers that based on certain rules
28:21
create this random yet realistic data. The important part though is that every time you run your tests you'll get different values compared to what you get with and there is a slim chance that Factory Boy might catch a bug for you.
28:41
If you're interested in this kind of stuff I'd suggest looking at the Python's hypothesis library. It's doing property-based testing and it takes the idea of random input to a whole new level. So the idea of the property-based testing came from a Haskell library called QuickCheck and hypothesis is the Python implementation.
29:05
You can use, so the way you do it, you use a given decorator which can inject randomly generated data to your tests. You specify what kind of strategies do you want, so in this case we're using a strategy text which is very similar to the
29:24
Factory Boy's providers and what it does once we decorate a function with the given function this test will run multiple times usually about 200 and it will test your function with
29:43
all sorts of different edge cases that usually that often easily slip away from developers attention. Since Django is an established way of doing web development in the Python world hypothesis comes with an extra module with support for Django.
30:04
Similar to Factory Boy we can create models populated with random data. To be noted with the hypothesis we always save the created model examples. So this is a real world example
30:23
of how hypothesis can be used for testing Django apps. If you look at the very top at the four inputs that we have there's one important import that is the most important and this is the test case. We're using the hypothesis test case to be able to wrap each hypothesis test into its own transaction rather than running all 200 of them in one single
30:44
transaction because then we'll get a mess. So let's remove the imports and focus on the test. This is a slightly stripped down example that I borrowed from a presentation made at Django local user group in London made by the author of hypothesis Dave.
31:05
So we are testing what we're testing here we're testing that if the project model instance can respect the limit of users a single project can have you can specify a collaboration limit on this project I just want to have three people and no more. So what we're doing
31:22
we are injecting one project model instance and a list of random amount of user models from zero to 20. Then we check if the project is at its collaborate limit and we try to add another user we expect an exception to be raised because we don't want to accept more users than we have
31:43
at our collaboration limit. Otherwise adding a new user should work just fine and when you run the test if I if hypothesis encounters that the test didn't succeed with a set of randomly generated data it will tell you falsifying example so in this case
32:02
we have a project with the limit with a collaboration limit of one and we've been able to add two users where the second one didn't trigger an exception which means there's something wrong you have a bug here's the catch with all of that tests are best kept when there is simple
32:22
and stupid and one should not have bugs in their tests with hypothesis tests can become more complex and you might even reach a point that where you will need tests for your tests which is ridiculous so when using um but when used properly and timely property-based testing can be a very
32:48
valuable tool so keep in mind that test code coverage is probably the most known and widely used metric so about a week ago I checked Django's test code coverage I ran it against
33:05
SQLite and I got 75 but here's the thing test coverage is generally deceptive metric if you have load code coverage then yes you definitely need more tests you're probably not testing your code properly but if you have high code coverage well it might be that your tests
33:22
are very good but maybe you just kind of wrote them in a way that you hit all of the lines of your code but you didn't for roughly tested it so unfortunately from in my opinion high code coverage doesn't necessarily imply high quality of tests we can do better we can do better with
33:40
mutation testing mutation testing is a concept that was introduced in the 70s and it involves changing the code of a program in some small ways like little tweaks and observing what happens when we run the test on this modified version of the code and this is a very powerful idea I thought it's it's so simple and and yet so brilliant in python mutation testing is
34:05
implemented by a package called mudpie and mudpie operates on the python ast to implement mutations so how does it work you invoke mutation mudpie from the common vine and you need to specify your target and your unit test the unit test obviously needs to test your target
34:26
code so if your target source source code has an if statement and the if statement has a logical operant and so you're saying if foo and bar then do this mudpie takes this logical operant
34:41
and inverts it into an all so our target code becomes a slightly modified version of the original and we call it a mutant now what we do we run the test on the mutant if the test failed on the mutant this is good we say that the mutant was killed by our test which adds points
35:03
to our final mutation score but if the mutant survived then our test isn't good enough it means that it doesn't really matter if we're using and or or the test pass anyway we must
35:20
we're probably not testing what we should be testing and mudpie can do a lot more there's a fairly long list of mutants it can replace arithmetic operants break continuous statements conditional operants constants logical operators and it can delete it can even delete whole conditional branches and decorators and do a lot more this is what a stripped down example of
35:47
a test run looks like you get a mutation score which is a percentage in this case 32.1 and it also tells you the total amount of mutants that it generated the number of mutants that were killed survived incompetent and timed out and i took i took this tool and i started
36:06
playing with the django test and see what results i can get so one pleasant trend is the django duration utils has a mutation score of 89 and its mutation score is 100 this looks
36:25
really nice this doesn't always hold true for example django encoding utils mutation score is 64 while the test code coverage is 100 so you can see that mutation score can actually
36:44
give you better insights into what how good your tests are django tutorial is excellent beginners don't need to search the internet for any video tutorials courses because if you go to the django official web page you get the django official tutorial and a whole
37:04
section of it is dedicated to testing because testing in the django world is an integral part of developing an app in my second year at university i discovered django it's also the first time that i wrote a test it was a very silly one i was just making a request to a url
37:22
and expecting a 200 response you know but you know why why i wrote this test is because when i followed the tutorial and i created my django app i also got this file test.py and that caught my attention the tutorial explained exactly what i need to do to get started with testing my code
37:44
so a testing framework is a powerful tool to motivate people to write tests and especially especially beginners django testing tutorial contains a section named when testing more is better it says that yes soon enough you're going to feel like your project is spinning out of
38:03
control because it grows so big and you have so many tests but you know just let it spin out of control it's fine let them grow when testing more is better there's a trap in there when with the number of tests growing your total test run time becomes slower and django
38:23
gives you all of the tools and encourages you to write very high level sort of functional style test so the speed of your test can suffer and this has other implications with slow tests you get a slow feedback loop with a slow feedback loop you get a slow development cycle sometimes if tests are so slow people just stop running them and that's why here are seven
38:47
tips on how to make your test not be slower than they actually need to be and number four will shock you
39:01
number one use a fast and less secure password hasher during your tests like md5 this is what django does for its own tests abuse simple test case when possible if you don't need the database if you're just writing simple tests use simple test cases brilliant use setup test
39:21
data if you don't plan to mutate any of the objects that you create in it because it's going to be called just once and here's the shocking bullet point number four use mocks everywhere and i'm joking i'm not serious just trying to emphasize that yes this is a joke there's
39:44
nothing inherently bad with mocks in themselves mocks are really good tool that you need to know how to use but if you're using mocks for with the intention of speed ups then there's probably something wrong in your app design and not in your tests mocks are really good for isolation
40:02
so use them for isolation be very careful what gets created in the setup method don't save model objects if it's not necessary and try to isolate your unit test from the rest from the rest of your tests and i would like to talk a bit more about the last three points
40:28
so last year i learned that this is called the tight loop so a tight loop like this where we create some model instances are red flags that need to be identified removed or reduced
40:45
instead of doing create how we did in the previous example what we can do we can create an in-memory model just like this just like this similarly an in-memory model can be created using a factory built method or you can create a stub object which is just a couple of
41:05
attributes using the stub method the last two are taken from the factory boy the caveat here is that you can't always do that for example when you have a many-to-many relationship you just have to save the model and finally isolate unit tests from the rest of your tests
41:24
your test suit may look like this it's a mix of you know unity unit-ish like tests a functional integration test and but it would have been a lot more useful and would allow creating a faster feedback loop if we separate unit tests from the rest because you can run
41:42
unit tests all the time you can even run them in a watch mode they're really fast they're going to give you they're going to give you the results instantly and then you can run all the rest of the tests less often one way to do it is to use simple test case classes for unit tests and tag them with a special label if you're in Django 1.10 if you're stuck
42:04
with an older version you can separate them in a different folder so that you can run them in total isolation from the rest of the tests even better you can make them simple unit tests or pi test which have no idea of what Django is and write more of them but don't neglect functional
42:25
and integration tests because obviously they are the one that validate the contracts between your units so you definitely should have both say your friend runs a little bakery business
42:40
and they sell freshly baked bread and other similar nibbles they hire you to help them build an internal app for them what the app needs to do is to help them figure out what should be the correct VAT value for their products and this will save some time for their account because taxes so we start with a basic naive schema where we get the product model
43:06
and the product questioner model where the one-to-one relationship between them the product questionnaire is the key to our VAT decision making it's a little add-on where we define basic yes or no questions and we build a decision tree to eventually land on a defined
43:23
answer is this product zero rated VAT or is it a standard rated VAT we'll need a view and we'll need a model form the view will display the questionnaire and answers will be saved on the product model for on the product questionnaire model for possible future
43:44
recalculations in the same view we'll converge to a final decision and save it to the product with a known VAT value it's going to be really easy to compute the final price of the product so I'll be showing example code for different approaches on how to solve this problem
44:01
where the very top we can have the production code and the bottom is going to be the test code solution number one in a view once we know that the form is valid we check if the product is a biscuit and is it coated in chocolate so in that case we're going to set the VAT to 20
44:23
this is a class-based view because of its brevity if you have an allergy towards class-based views you can just translate it in your mind into a function based view as for the test what we're doing we are creating a product object we are making a request
44:41
by passing some data and then we are refreshing the product from the database and we're checking if the VAT equals to 20 in reality we understand that the if-else is going to be much bigger we can split it into multiple methods but the essential part is that the entry point of this calculation is going to be in the view and for each branch
45:07
we're going to have one test and each test will do the same thing it will create a product object and it will make the request and it will check this VAT value and they'll be they'll be they'll be just repeating ourselves and we're going to do it for
45:23
biscuits and bread and flapjacks and other items so to test each one of those items and to calculate them to check their VAT value we will also be testing the route the router will be interacting with the database to eventually send an input to receive an output
45:44
and if you're thinking that you can remove the database interactions with the mocks I think mocks are not a solution in this case because we can do better solution number two the conditional check gets moved to a more appropriate place the form
46:03
so in the test we're again creating a product model we are instantiating the form we're passing in some data checking if the form is valid we're going to save the form refresh the product from the database and check if the VAT equals 20 to test it we again need to create a product
46:22
object but we know long but we are no longer testing the routing which is an improvement over the solution number one by the form we send an input the answers to the questions to receive an output third and last solution moves the entire decision making to an old boring python class
46:44
or a function if you want to it has a method called calculate VAT that does exactly what it says and in the test we're instantiating the VAT calculator calling the method with a bunch of arguments and we're expecting the result to be 20. somehow we forget that Django is a web
47:04
framework not a programming language different constructs can live outside of core Django components the VAT calculator has no idea what Django is and it doesn't need to when testing we're not testing the router we're not testing the ORM all we do
47:25
was sending an input to receive an output and what we did is called decoupling the coupling has various benefits reusability as a developer I can pull out the VAT calculator and build a separate app that calculates VAT and helps others maybe figure out their VAT values
47:46
extend stability if my friend decides to expand their little business to a new country and this place has different VAT values it's going to be much easier to update them in one place in the VAT calculator rather than going for all of the views and forms
48:02
and trying to update it there and finally to stability you saw how easy it was to test the VAT calculator this approach of isolating business logic allows creating more and faster unit tests and keep the functional and integration tests at a manageable rate
48:23
tests have more in them than we think a test has the capability to drive a better app design in terms of Django it's about moving complex logic away from any Django related components if it's a complex behavior that isn't directly tied to the database
48:41
templates forms or views then build objects that perform those complex actions and unit test them thank you very much for your undivided attention and happy testing everyone