Add to Watchlist

Mashing up py.test, coverage.py and ast.py to take TDD to a new level

20 views

Citation of segment
Embed Code
Purchasing a DVD Cite video

Formal Metadata

Title Mashing up py.test, coverage.py and ast.py to take TDD to a new level
Title of Series EuroPython 2015
Part Number 18
Number of Parts 173
Author Arpas, Tibor
License CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
DOI 10.5446/20212
Publisher EuroPython
Release Date 2015
Language English
Production Place Bilbao, Euskadi, Spain

Content Metadata

Subject Area Computer Science
Abstract Tibor Arpas - Mashing up py.test, coverage.py and ast.py to take TDD to a new level Users and developers especially, hate waiting. Computing has adapted and we almost never wait for the computer for more then 10 seconds. One big exception is runnig a test suite which takes MINUTES on many projects. That is incredibly distracting, frustrating and dragging the whole concept of automated tests down. I present a technique and a tool (py.test plugin called "testmon") which automatically selects only tests affected by recent changes. Does it sound too good to be true? Python developers rightfully have a suspecting attitude towards any tool which tries to be too clever about their source code. Code completion and symbol searching doesn't need to be 100% reliable but messing with the test suite execution? I show that we can cut test suite execution time significantly but maintain it's reliability.
Keywords EuroPython Conference
EP 2015
EuroPython 2015
Series
Annotations
Transcript
Loading...
thanks and the thanks for coming everybody so that that's the the talk matching up by this college by and it's still
part to take TDT to a new level so let me say just a
quick things about me I've been a freelance programmer since the beginning of my career like 20 years ago a couple of times I took responsibility for the whole project and had a few superconductors to the library I chose 1 1 project that I chose so vital for the delivery it was in 2008 and I didn't have to touch anything else is then fortunately at the moment I have 5 patterns subcontractors in 1 of his in Bratislava working for 1 client on a long-term project and looking for more clients OK I would like to have a little survey so server time so how many of you did the right at least 1 Automated Test called
only to who has tests you'd longer than 2 minutes a whole lot longer than 10 minutes OK so some some some hands that the no OK 8 dollars OK 2 2 hands here I was getting a broken builds to that's great was using now about by this what is that thing so the last 3
user interaction limits 100 ms is designed for use it to feel the system is reacting immediately 1 2nd is the limit for users fall fall to stay uninterrupted even though they not to delay and the 2nd is doing it for years until they think the system is broken and start doing something of so computed in adapted developers have done a good job in making a joke recurrent recent years there is very few activities in today's computing where you have to wait for more than 10 seconds on
but how about is a competing interests you it takes because
minutes our so it's 50 times slower than most of the other computer tasks in brings terms load on the computer delays with artifacts and screening so what are the consequences little
presented written more than anybody else as executing test best center fuse the verb false so much it's no wonder then that under pressure or destruction some developer doesn't from the person doesn't know what is negative results and come into failing built which makes the lives of developers and 1 difficult and sometimes starts downward spiral of broken
pursued means so better life increases developers valuing using and maintaining pursue the most a most punished and raise the most time actually think of that but that's the execution time is the single biggest flaw of automated tests idea as a whole but
how about done just affected
majority of coaching is at local so it is based around the whole each time there is of
course a solution most of us have used to develop a things i'm changing just is not so it's just execute the related tests however it's quite cumbersome and unreliable like being correct in his hands became quite the dependencies in like this and also
and 1 after 1 of the properties of 1 of the purposes for purposes of this to to discover the failure of each new didn't think of being able to call that the change sides and that implies that something you wouldn't when think of so let's explore the
idea of affected and not affected and unaffected tests on a very simple project comprised of 1 pattern file let me have your most attention and please have a look at the scandals project and try to grasp it completely for those who were you don't would unknown patterns discovered announcing any methods which are called best underscore something things of and so this constitutes a valid tests you including the the called common
test and here we have a grid of dependencies between tests and methods of partcular project whatever you do inside the subject method body there is no chance to influence best underscore and the source code
again you can have all the inside subject and best underscore at is not going to be influenced it never calls subject for test and the score it at to start calling subject to any of the methods it's already going have to which will trigger right execution and created and updated dependency metric so
back to the dependency matrix out of 6 positions we have 4 crossed once so almost all of them but on bigger projects erasure is going to be much smaller so there's a lot of methods so that can be changed and only implant small issue maybe suspicious about so how could we checked metrics and dynamic language like they might have and member feeling the same way when I when hearing about college recording for the 1st time I thought that it would be fragile so on the on the reliable but now it's very good to stable and widely used project on creating a metrics on the slide this just a little addition to coverage by itself it is the same image obviously at the moment it doesn't vary across technologist text you can kinetic execution of C + + code or transcripts figure from your from part of it also doesn't record data failed but file changes if that input of your tests and it changes the execution path you get from results but for the circumstances where it works so iteratively I wanna show
you a tool which automatically executes only affected by some members of change the acutely I changed my mind to correspond to so if you are the you with you can join the 2 methods so it was screencast so that it doesn't go wrong in the presentation that I would have it so that when the right everything went as expected so the
idea transferred into a pool and so on best known practice this morning besides intestinal up or contagion could help and let's go briefly through the library which is which of these based on coverage by it's a giant which
which allows all these after some initialization and executing of this continent but again the filings and 1 numbers of executing called College buys mostly used this command line reporting tool of the features of courage by which this non users are almost the and almost part of the API it doesn't sound very good but really there is just a little to underpinning to attribute used and also that bachelor was alpha of coverage by reached out when we started with best known that you would like to know and if there is any obstacles and and probably also fix a fixed and if there's something that makes problems and also man would like to add the coverage by the functionality the features to track reach which best serve which methods are actually executing which lines is it is it exactly the information which we are tracking and In this way with convergent there could be a tremendous so after executing teach this style of its college we're getting file names and lines of codes triggered by the best From there we need to get methods which are executed and the AST from standard libraries schools with tool to do that that was my 1st contact with syntax tree you know when I I implemented so the from the many can imagine what they are this I would say there are much more they are not so much more started out of things in programming so from me diverse obstructer's actually quite distracting that's 1 only needs to parts the source code and understand it enough to know where the land boundaries of methods what they are so as the library is a little work your authorities but also be but it's ready and works in all regions of vitamin also future once and I was like to learn the basics of if you would like to bring up a few of those be interest in learning KST a little bit more common the 2 resources so there to study on the obviously the best answer so by this worrying so far the spiders seems to have a more more active community recently and also more but here I was surprised to to see more hands raised when you ask about the about which we show this this you use but the final I 91 and the thing there's is a lot of projects using them also tend to not switching anytime soon so I'm interested in all supporting self-assessment and also and the of anybody else experience in doing some of this stuff and also on would the left to talk and so maybe I'll be need help me that also and 1 interesting aspect of the whole project of the also that I call that the this is a valuable tool and which many people could and it would save them a lot of time so I asked for for money and on Indiegogo and I didn't have any Fuller's or blog or or any other open-source tools so I used before or and that will be 4 so it was really difficult to get the word out but still owner the mortgage amount of fluid that collected so that it was something that was a nice despite think and I would encourage you if you have if you have done something good for the community and have thought of long or words and would like to dedicate your time to to do some valuable to tool that don't hesitate and then use this root I think that there will be more market in the future so of the they a little sneak peek
into the into the future I hate the way tests used results are presented on the console hate scrolling chronicles along time and the reasoning through the stack traces so I was thinking about some better way to errors and I think the best way would be the best I can think of so far would be to present the heiress inside of text editor you use on so something similar to to for example and this is a screenshot from my from 1 I think it is a very good to Michael way and the easy way to and only volunteer subculture career and in the combination with the interactivity of of best where you get and you can get the results really quickly and I think this is this is the way for the future of tool to make it as an to of course is not specific to to test on it then this would apply to any test China but some thing it in the combination of the quick feedback and looked the presentation would be great their so
this is the conclusion and this 1 is also use please give me feedback share and tell your colleagues and so this is conducted on and it's time we have enough time for questions so I know there's no many and this is ahead of you guys of questions over the microphone to you the moment but we should the k below is the only way to force some cholesterol around all the time like for example I
have also does so that just so that the Beast sugar and I wanted to run all the time Canada not yet but it so it doesn't have many features and so on so let's see how it goes on until also kept it and the minimal so that it's easy to study and to prove the concept really works and 2 of them and not in any features is like the next step and go out of hand you should test long-running kind of this of this a process and monitor much of files that can it also produced a report that I can process a milder does have an API that I can use to tell me how old source code source code lines map to tests that I can use information myself I'm not but I guess that's the only go that's the addition to call each that part I mentioned like because if you have some I don't know what would be maybe can can again and the microphone back what what would be below used or maybe ever talk about it but maybe you can tell that use you can use case I have in mind was for my liking talk yesterday that the mutation testing right 1 of the stumbling blocks from as it takes a long time to run all the tests inside this these loops so if you can determine exactly the test you want to run based on some you know get differ something like that then you can drastically reduce the runtime for these kinds of things make practical so that would be where something like this could come in really handy for that kind of work as well as warning I I I can't see how I'm strand untrodden envision how I can use this to you have to make to make myself work better yeah but I said that there is not many features and there is no API so but it's really really small tools so it would be easy to add any of the but at this point the I of how do you test test the question is what this suit but I have done that there is a problem in calling coverage by cruising and so on I can have a pursued but Kennedy is applied in itself which is a little bit of about 1 of but the earlier there is also solution I could on on on the parts which are which rely on the cover each other the next feature would be to to testament to adverb be to manually specify methods and tests with the manually specified of the dependency so that the even if the tracking is lost In the recursion scientists to work and thus attract as changes that you make in and by the test features as well to few data fixture with that a spot from well if its data files features if it's and Jason and like was stronger fixtures right now that was 1 of the things I mentioned that it doesn't work when a across the chromosome technologist specs and it doesn't work then you have data file input or some inputs from external services and so ideally you would have pursued for each constructs everything in pattern called also the features then is of then it's then it's Python code and extract execution as quickly they could go back to the little example if they would lose subtraction addition stuff yeah uh order did not show in medieval string comes was awarded substrate methods deemed a call to add well how would reduce obsolete as new depends visited short form and function grace How would have no that would know that after running test once or when would notice if you add the coal and my to subtract metal get right the subtract manifold is called by to test right so all of them would be both of them would be murderer executed the perceptron and the support it has Edward VIII and small new dependencies it wouldn't be and it wouldn't again the dependency also 2 dozen new dependencies because subtract calls ad but best doesn't cold subtract arcade got its target muscles thanks also that celebrity
model like that of even using the test on on the 1 project and most of the body reports have been like this unfortunately it so we found out that this 1 is applied to each you which quite the quite a good surprise because because and also was afraid it's going to be fragile and stuff about the biggest problem actually is that so this are dependent on independent even if you don't know about it and there's some fixtures which and that doesn't have that we initialized or something like that so what then you get failures because this small and you always run some order dynamic subset of your pursued which doesn't happen and many many other circumstances so so that's like the biggest problem in adopting next question it is my test cause a helper function is my test help functions and the help of function changes within yeah this is the part of the trade yes no it's not a part of the tree part of their execution right well the the syntax tree or the radius the passing and only used at the end of the hall analyzes and it's not that important but the important thing here is that when you are on the test and you that this term cause the helper functions it seemed the least of executed lines and executed methods that of the of the best of and it's in the dependency matrix it will appear as a dependency so that probably answers versus question which was about applied to speech is not about data flows about text files high test fixtures in the Netherlands is a question about staying in the Python well no question about Jason fixtures yeah and that that answers to questions yet msec is these shows are are created by a method to right so so the it's going and this is how does it we start the test for analyzing it is that we start the entire process will this is like a critical forests and investment is implemented as a pilot test like in and so how does it restart the test when it tries to renew tests as in how does that affect the sessions scoped features do we start the entire process all the down to start with well I I'm not sure that about the session efficient sculptor of pictures and it really depends on the changes in or are you asking about the runtime itself like I can about about the feature set up and then down based set up after his when a couple of test attend all the fixtures and restart again next testable amateur this might be mm-hm well not tested yet I'm not sure how it would behave so that a session fixture always around venison currently always and just since the beginning of the test execution time and if you change anything in there that's probably that's not distract because the specific test in doesn't execute any of the social feature right so that wouldn't wouldn't get called the need to registered as as a dependency dense incorrectly that is the function does something nasty for example change something global well you and other function depends on it on this global belly so we can attend bold can break the function which you ended read the tests will not and followed it the other functions for the all it is quite difficult to explain but found in if that evil function changes in global value FIL appear in the dependency matrix idea it's called by test or it's not for every single test you can set and full the test which do not call the function it doesn't matter what the EU will function to the broken exploding that function but the tests are not executed but this did not execute use it for example if if my function depends on the and state the tangent function a depends on the state and then have function B that starts to modify the state and makes in runtime makes the original function right not that I don't that this is the best of dependency and because the evil function gets called in just 1 and the best to realize on that on that and the global value so that's what I called this the tendency in that talked about that would be a problem but not if there is you know if it's the tests are independent and that this tool doesn't rely on that value and 3 sets it again or doesn't use the value it doesn't matter it gets registered oracle repeat it works as it yeah fact that seems to be at
Statistical hypothesis testing
Multiplication sign
Projective plane
Moment (mathematics)
Archaeological field survey
Client (computing)
Mereology
Programmer (hardware)
Computer animation
Energy level
Dependent and independent variables
Pattern language
Right angle
Library (computing)
Statistical hypothesis testing
Process (computing)
Software developer
Interactive television
Limit (category theory)
Physical system
Recurrence relation
Suite (music)
Computational number theory
Structural load
Multiplication sign
Software developer
Division (mathematics)
Spiral
Statistical hypothesis testing
Statistical hypothesis testing
Computer animation
Term (mathematics)
Uniform resource name
Pressure
Task (computing)
Run time (program lifecycle phase)
Statistical hypothesis testing
Arithmetic mean
Video game
Error message
Computer animation
Software developer
Software developer
Multiplication sign
Statistical hypothesis testing
Statistical hypothesis testing
Suite (music)
Multiplication sign
Statistical hypothesis testing
Statistical hypothesis testing
Category of being
Mathematics
Computer file
Projective plane
Pattern language
Statistical hypothesis testing
Source code
Projective plane
Statistical hypothesis testing
Statistical hypothesis testing
Statistical hypothesis testing
Slide rule
Spacetime
Computer file
Presentation of a group
Code
Multiplication sign
Dynamical system
Mereology
Formal language
Medical imaging
Mathematics
Matrix (mathematics)
Cache (computing)
Kinematics
Addition
Computer file
Projective plane
Moment (mathematics)
Statistical hypothesis testing
Syntaxbaum
Metric tensor
Computer animation
output
Figurate number
Form (programming)
Row (database)
Web crawler
Coefficient of variation
Multiplication sign
Source code
Home page
Mereology
Table (information)
Pointer (computer programming)
Network topology
Row (database)
Website
Arc (geometry)
Metropolitan area network
Musical ensemble
View (database)
Bit
Functional (mathematics)
Statistical hypothesis testing
Statistical hypothesis testing
Abstract syntax tree
Maxima and minima
Content (media)
Uniform resource name
Quadrilateral
Convex hull
Wide area network
Personal identification number
Mapping
Computer file
Open source
Line (geometry)
Disintegration
Abstract syntax tree
Emulation
Number
Goodness of fit
Fluid
Root
Computer programming
Authorization
Boundary value problem
Gamma function
Conditional-access module
Mutual information
Metropolitan area network
Alpha (investment)
Rule of inference
Plug-in (computing)
Raw image format
Standard deviation
Dialect
Information
Projective plane
Computer program
Code
Core dump
Coma Berenices
Machine code
Line (geometry)
Word
Computer animation
Blog
Units of measurement
Library (computing)
Statistical hypothesis testing
Presentation of a group
Feedback
Multiplication sign
Feedback
Moment (mathematics)
Combinational logic
Tracing (software)
Statistical hypothesis testing
Physical law
Text editor
Video game console
Error message
Resultant
Statistical hypothesis testing
Suite (music)
Code
State of matter
Multiplication sign
Scientific modelling
Source code
Insertion loss
Mereology
Fault-tolerant system
Subset
Mathematics
Matrix (mathematics)
Forest
Recursion
Oracle
Covering space
Process (computing)
Constructor (object-oriented programming)
Functional (mathematics)
Abstract syntax tree
Message passing
Network topology
Order (biology)
output
Pattern language
Right angle
Point (geometry)
Trail
Service (economics)
Computer file
Perceptron
Manifold
Dynamical system
Online help
Trigonometric functions
Statistical hypothesis testing
Causality
Term (mathematics)
String (computer science)
Subtraction
Traffic reporting
Form (programming)
Run time (program lifecycle phase)
Addition
Information
Projective plane
Line (geometry)
Set (mathematics)
System call
Loop (music)
Radius
Personal digital assistant
Speech synthesis
Loading...
Feedback

Timings

  578 ms - page object

Version

AV-Portal 3.8.0 (dec2fe8b0ce2e718d55d6f23ab68f0b2424a1f3f)