Python Table Manners: Cut the Cookie Gracefully
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 130 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/50097 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 2020112 / 130
2
4
7
8
13
16
21
23
25
26
27
30
33
36
39
46
50
53
54
56
60
61
62
65
68
73
82
85
86
95
100
101
102
106
108
109
113
118
119
120
125
00:00
HTTP cookieBitMusical ensemblePerfect groupComputer animationMeeting/Interview
00:36
SoftwareComputer multitaskingSoftware maintenanceSoftware testingData managementTask (computing)Military operationInformation securityTemplate (C++)HTTP cookieComputer fontSource codeVirtual realityHash functionSynchronizationInstallation artSineLibrary (computing)Standard deviationExecution unitSoftware frameworkPlug-in (computing)MechatronicsPersonal digital assistantConfiguration spaceObject (grammar)DreizehnConsistencyError messagePhysical systemSpacetimeCoding theoryCodeOperator (mathematics)Electronic mailing listInheritance (object-oriented programming)Casting (performing arts)Functional (mathematics)Online helpHash functionSynchronizationRevision controlType theoryVirtual realityTable (information)Software maintenanceProjective planeComputer programmingPattern languageComputer fileSheaf (mathematics)Unit testingSurfaceMultiplication signJava appletArrow of timeMoment (mathematics)Cartesian coordinate systemServer (computing)Object (grammar)Web 2.0Interactive televisionService (economics)Group actionProgrammierstilAsynchronous Transfer ModeExterior algebraSoftware testingConfiguration spacePlug-in (computing)Radical (chemistry)ConsistencyIntegrated development environmentSoftware engineeringFlagString (computer science)Directory serviceLine (geometry)Grass (card game)CASE <Informatik>Cellular automatonProgrammer (hardware)Graph coloringPiMulti-core processorSocial classData managementCodeExecution unitSpacetimeLibrary (computing)Parameter (computer programming)HTTP cookieStandard deviationEmailError messageAdditionReal numberView (database)Visualization (computer graphics)Formal grammarCommitment schemeInstallation art
09:23
Configuration spaceExecution unitEmailLogical constantCodePattern languageParameter (computer programming)File formatData typeError messageVirtual machineElectronic mailing listExpected valueExtension (kinesiology)Computer fileComputer configurationLocal GroupLibrary (computing)Standard deviationPauli exclusion principleTask (computing)Event horizonScripting languageInstallation artGastropod shellSoftware developerMilitary operationOperations researchLocal ringType theoryFormal languagePhysical systemAdditionPositional notationLine (geometry)Square numberDirectory serviceString (computer science)Ocean currentMultiplication signDot productLevel (video gaming)Compilation albumFluid staticsCommitment schemeCellular automatonHookingVirtual realityNeuroinformatikRadical (chemistry)TrailComa BerenicesDefault (computer science)Buffer solutionComputer fileComputer configurationField (computer science)Software repositoryOperator (mathematics)Local ringSoftware testingSoftware developerScripting languageGastropod shellType theoryComplete metric spaceTraffic reportingStatisticsRun time (program lifecycle phase)Projective planeUnit testingProgrammierstilGroup actionConfiguration spaceLibrary (computing)Electronic mailing listFunctional (mathematics)Revision controlComputer programmingRootParameter (computer programming)File formatSpacetimeMaxima and minimaWeb pageExtension (kinesiology)Virtual machineCartesian coordinate systemOrder (biology)Arrow of timeTask (computing)CodeData managementZeno of EleaIntegerPi2 (number)Mobile appNamespaceLink (knot theory)Demo (music)Content (media)Patch (Unix)Automatic differentiationBooting
18:09
Software repositoryConfiguration spaceTrailRevision controlMenu (computing)Correlation and dependenceSpecial unitary groupCodeMaizeNormed vector spaceCloningDivisorData typeDenial-of-service attackIntegrated development environmentVulnerability (computing)FreewareInheritance (object-oriented programming)Template (C++)Data structureComputer clusterCASE <Informatik>TelecommunicationIntegrated development environmentLoginProjective planeLeakMathematicsFlow separationSoftware bugAnalytic setSingle-precision floating-point formatComputer programmingMathematical analysisVulnerability (computing)Template (C++)Configuration spaceLibrary (computing)Revision controlDatabaseMessage passingDifferent (Kate Ryan album)Data structureSpring (hydrology)Coma BerenicesSpeech synthesisComputer filePhysical systemType theoryInstallation artSpacetimeRepository (publishing)Line (geometry)User interfaceMultiplication signHTTP cookieData managementTask (computing)MereologyDefault (computer science)Information securityCodeKey (cryptography)Functional (mathematics)Uniform resource locatorGroup actionTable (information)Electric generatorRule of inferenceSheaf (mathematics)HookingException handlingSoftware testingGoodness of fitPatch (Unix)Commitment schemeVideo gameStatisticsOpticsPiMedian
26:55
Software testingMessage passingTotal S.A.System callTable (information)Moving averageCommitment schemeMultiplication signCode refactoringPerfect groupBlock (periodic table)Software bugStructural loadWeightPoint (geometry)Computer animationMeeting/Interview
Transcript: English(auto-generated)
00:07
Perfect. So it's a pleasure to have you here. And you're going to talk to us a little bit about cookie cutter science as well, right? Yeah, you're definitely right. Cool. I am personally interested in this talk as well. Because I really like Kedro
00:23
and Kedro is like one step after the cookie cutter data science. So it's inspired on, but we can have a chat about this later on. So yeah, the floor is yours. Take it away. Okay, then welcome to my talk Python Table Manners and Color Cookie Grassfully. Yeah,
00:43
I'm Weili. I'm a software engineer at Rockton Slides and also a volunteer at PyCon Taiwan. As you can see, this is a teacher for help from PyCon Taiwan. Also, I am a maintainer of commitment tools, which is our tool I mentioned in this talk. Today, I will illustrate how
01:01
to first clean out your table before you get to your dinner. And we'll ensure you that you'll put the correct tableware on your table, then we'll learn how to use this tableware elegantly. Because the step may be too trivial for you, you'll need some mnemonic press. And if you're asking others, like to get to help you, you should
01:24
say please, in such an occasion, you'll like to speak formally. And when we have our knife to cut a cookie, we'll ensure our own safety. And the last step we'll see first the cookie. Starting from dependency management, this might be how we
01:43
used to start a Python project, we create a virtual environment, and then we activated. And after that, we'll free some packages into requirements.txt. But sometimes we just forget to activate the virtual environment, or we forget to add a package into requirements.txt. So we can use tools like Pippin, because Pippin can
02:07
management, manage your virtual environment and packaging at the same time. So you no longer need to manually sync up your virtual environment and your requirements.txt. It's owner also generate hashes from the package choose Pippin download from type
02:23
PI. So it's can ensure you that you can get the same packages next time you installed from it. You can initially visual environments through Pippin install. Then this is how an empty PIP file looks like. Pippin views PIP file and PIP file
02:41
.loc to manage dependencies as an alternative to requirements.txt. And the API is pretty much the same as PIP. So you just need to type Pippin install, which package is equal to which version. If you add a request to your virtual
03:00
environment, it will update a PIP file like this. And it will add this, this section into your PIP file .loc. This hashes is generated from the code Pippin download from PI PI at the last moment. Even if the next time you download it,
03:20
the code is changed, but the version is still 2.22.0. Pippin will rise and error so you're guaranteed to have the exact same package next time. But sometimes you just don't need everything in your production environment. So you can install the packages into your development
03:40
environment only through adding a dash dash def argument. And it will appear in the def package section in your PIP file. And because we already set our virtual environment and manage our dependency at the same time, we need to run our Python program inside our virtual environment. You can do so
04:01
by Pippin run Python your program or something like Pippin run Django management the PI to start the web server. But some people might say that Pippin does not update that frequently or it's just updated two months ago, and the back of it is really slow. And it does not sync up
04:21
with install request instead of the PI. Maybe you could try poetry. The concept of poetry and Pippin are alike. So I'll list the command here as a reference. For releasing a package, I will recommend you using poetry because you don't have to manually update dependencies on both
04:44
PIP file and the other PI, poetry will do that for you. But for Python application, I'll say both poetry and Pippin work for me. Testing Python comes with standard unit test unit test framework in its
05:00
standard library. But it's borrow the concept from J unit in Java. So today I want to introduce pytest. Why should we use pytest? Because pytest is considered to be more Pythonic, and it's compatible with the old unit test style. And in unit test, you will need to use a
05:22
assert function like assert equal, assert true or assert false and extra etc. But in pytest, you just need to memorize asserts. And the other side, you can use the same syntax as the normal Python you use. And pytest provides better test discovery, advanced
05:41
features, and it also comes with plenty of plugins. This is how we pytest. Actually, after we install our virtual environment, we should always install our packages into our virtual environment and run our Python program inside the virtual environment. This is how a unit test unit test style test
06:05
looks like. First, we use the setup to prepare all the data needed in our test cases. But because unit test is borrow the concept from J unit, it's set up function is camel test, which consider which is considered to be non Pythonic, and we'll
06:23
inherit unit test test test class. And as I mentioned previously, we'll need to memorize the assert that assert functions. And in pytest, we use fixtures to prepare individual data for individual test cases. So we no longer need the
06:43
setup function. Also, we we don't need to inherit a best class. And we don't even need a class to run pytest. And the assert function become much easier to memorize because you only need to use assert and the same test afterward is just
07:05
the same as how you use Python daily. And this is my configuration for pytest. I use pytest dot init to configure pytest because style dot config is not recommended for pytest configuration nowadays. And after pytest 6.0 is released,
07:25
you can even configure pytest through pyproject.toml. These are the plugins I use in almost every of my Python projects. You will use pytest mark for replace the object letter how to test like if you your program interact with AWS
07:44
GCP or others, third party services, you don't want to actually interact with them because it's real cost you money. So you will want to use a fact object to to be testing your unit tests. And pytest coverage can show you
08:01
which portion of your program is not covered by your unit tests. And pytest dot xdist can accelerate your test by distributing your test to multi core coding style. As Python programmer, we not only want to write correct code, we
08:24
also want to write elegant code, we can do so by flake. Flake is a tool that can enforce style consistency across your Python project. It can also check possible error before you actually run your program, and also eliminate the bad coding styles. In this example, I
08:44
redefined the OS library as a string, which could be a possible error because after this line, you can no longer use methods like OS dot get current working directory because OS now is a string. And I add an additional space
09:04
here, which is considered to be a bad coding style. After running flake, it will tell you where are the errors and best best mouse. This is my configuration for flake. I use that on the config. In this section, I will introduce
09:21
you a lot of tool that relates to coding style. So by following this configuration, all these tools will not conflict with each other. Piling, the functionality of piling is pretty much the same as flake, but it can generate more detailed reports. This is a, we use the
09:43
same code and run pilings. This is the report, it's just me. And if you run pilings with dash argument, it can generate a even more detailed report, which you can compare with your previous piling run to see where your coding style is improved in between the two
10:03
different check. And I use pyproject internal for configuration for configuring pytest. I used to use pilings RC for configure pytest. I follow that the default pilings RC contains too many default values, which
10:22
is distracting when and Maggie make me hard to find the thing I really want to configure. So I think I've seen pyproject eternal or you can use that config to configure pilings, which might be a better solution. My pie in Tyson communities type annotation is now
10:44
encouraged. So we now have tools like my pie to do static static type checking. And by doing so, so you can avoid possible runtime error because my pie can run compile time type checking. And by doing type annotation,
11:01
it's can enhance your readability. Now, it's not only work as like dark string, it's machine checkable documentation. In this example, values is annotated as a list of stir, but we passing a list of integer into, which will be a runtime error. But tools like
11:25
second and pilot won't, won't, won't you about this error, but my pie will, my pie will tell you that you should actually passing a list of stir instead of list of integer. The first argument indicate that you want
11:43
to check all the file with the extension. And the second one will ignore the arrow. That's your third party library is not type a note added. Because what we actually care about is where our code is type a note added. And this is how I configure my pipe through
12:02
style that config. Let's do nice. The configuration, the other arguments I showed in the previous patch. So after this, you no longer need to type arguments. We can even take one step forward by fixing the style automatically to black. Using black is really easy. You
12:22
just need to run that. And it will fix all the styles under your current work directory. This is how that reformat the red one is the code before black actually fix it. Because backslash is not recommended in our day that will use a square to do the black line. And it will
12:46
fix the notation and also add an additional space between command and its command and its content. Why should we use black? Because the black code style is not configurable. You cannot tell that how to how to
13:03
format or I don't like this portion of black style. No, you cannot, you need to establish the black code style for you. For your whole file, you can not even get it temporary ignore mark to format only a portion of your
13:23
code, you need to format the whole file, and which leads to no more argument about which style is better. So you can focus on what really matters. The value feature you want to deliver to your customers. And there should be one and preferably only one obvious way
13:44
to do it. It's from the demo Python. And this is my configuration for that. This is not for for black formatting, it's just to tell that which file is you include, and which file is should exclude. I thought in
14:03
this, in this Python file, I randomly import some libraries. But according to paper, we should start our libraries in the following order, we should first have standard libraries. Second, third party library and serve our local application or library imports. And you
14:23
should add a blank line between each group. After running, I thought it will group this import for you. In addition to group is it will start the libraries alphabetically. So the next time you want to buy, which which library is imported, it will be much easier to find
14:44
than randomly sorted. And this is my configuration for I thought this is all the coding style related tools I use for my Python project. And this outcome then for formatting and linking. But it's just way too many
15:02
comments. So we need some tools like pipe invoke for task management. It's like a max file, but it's written in Python. I'll demonstrate how we use invoke in report generator command line in practice. This is this
15:23
is how we install our analyzer and run unit test before we use invoke will need to memorize this long comment. But after invoke the comment become much shorter. But you might say, even even if they are much shorter, now we still
15:43
need to memorize that. But no, you don't. Actually, you the thing only thing you need to memorize is involved dash l, it will list all the comment you implement. So how can we implement this comment. So test that by your app this file to the boot of your Python project, and
16:03
then move the command to the tech. And best of all is can ensure that your Python program is run inside your virtual environment by adding virtual environment prefix in your test apply. After you introduce a bunch of tests, you
16:25
might want to modularize them through the concept of namespace. So as you can see now, the comment becomes invoke build dot develop. But with your comment become even longer now. So now we need auto completion, invoke, invoke
16:41
comes with a completion script, you can generate the script for each of your shells through this comment. And after that, you can type in both built that tab, and it will show you all the options you can choose from. So why not just use Mac file? Well, because we are Python developers, and some
17:02
tasks might not be easy to handle through shell script. And just good in differential command might be different. And in Python book, you can combine the power of Python and shell script. It's a best of both. Actually. And because
17:22
people might forget to do a check, even after we made a check much easier, so we can ask it to do a check through pre commit. So how do we commit do a check for us? For coming can run some command before we do any get operation
17:40
like it's push and get commit. We'll first need to tell pre commit, what command we were to run. And you fix example, I first use a rebel local, which means it will run local command on your computer. And the first hook is style
18:02
reformat. I will check it at the stage commit, and I will run invoke style reformat at this stage. The second one is style check. I will do style check when when I do get push. There are also some existing hoops that is commonly used. So
18:22
becoming has a repository becoming hooks. For example, I have introduced an optics and file fixers and traveling widespread, which will remove the traveling widespread in your files except my phone files. And popular project
18:40
like that I thought and you won't like it has their own hooks on GitHub as well. So after we configure it, we need to install it into your local local repository. So pre commit install, because I mentioned I will use stat push and commit. So I install hook type pre copy and pre push in
19:04
this example. After that, you do get commit, and you will run the end of life fixer, trim the white space, and do the style reformat. And when you want to push your code to your remote repository, it will do a first two and then we'll do
19:22
style check without style reformat. Speaking of it, we might want to cultivate a git commit convention. If you're like this guy, and you will see it look like this. And it will make it's really hard to find the right version to roll back
19:41
to when you your system goes down. Because all the commits update, you cannot distinguish them. So commitment tools is here to help by using the commit, which is a comment from committed said, you can you'll get a user and user interface
20:03
that will first ask you which type of change is this one. And it's also gives the user a hint that you should not make a lot of different different changes into a single comment. For example, you should not add bug based feature and refactoring into a single comment with which will
20:23
make it really hard to review. And then it will ask you about which scope is it the subject of it, and whether this comment is a breaking change. And you might want to add some some more details to your to your comments. And then you
20:44
might want to reference to your GitHub issue, GitHub issue or don't you rotate it. This is a comment we just generate your previous patch. And if you're continuously using commencement, you'll see this kind of good luck, which is much
21:01
readable than update. Committances also comes with some advanced features, it can prevent you from not using commencement because people still forget to use commencement and they use it to come in with update. They sometimes and because the rule we just use is conventional commit, we
21:24
can use customizable commit rules. And because our coming messages standardize, we can also bank our project version and generate change your own. So come in to send it through the functionality of committee set. And best of
21:42
all, I'll hold a spring tomorrow and Sunday. So that spring, join us on virtual run 10. Security issues, you might have seen this kind of warning on GitHub, it tell you that some of your libraries might have security issues
22:01
that's usually upgrade your your library to certain version. You can do so locally through safety. You can check the vulnerability by running safety check. In this example, it tells me that pycrypto 2.6 2.1 might be
22:20
dangerous. So I need to update to a higher version. But if you are using pipping, you can just run pipping check. Safety will search your vulnerability in the CVS database for for the known vulnerabilities. Embedded bandit can do static analysis
22:45
on to check common security issue in your Python program. And in this example bandit tells me that in this portion of my code, I have a median severity security issue and bandit this is a how confident bandit think this is
23:05
actually an issue. And this is bandit will also tell you how why this might be a problem and how you could fix it. But not all the warning should be fixed. In this example, it tells me that I should not use a cert. It's
23:21
because a cert might be ignoring some Python configuration. So if you are using a cert to identify your user in your logging system, you might be a potential vulnerability. But we still need to use a cert in our test cases, right? So you should add a test into your escrow section into your bandit configuration. But if you are
23:46
the whole file, just some section or some lines in your code, you can add no secure after the end of the line of code you want to ignore the warning. So cool. Let's talk about cookies. Project template, you might want to
24:02
use all these manners in your all your pricing projects afterward, but configure it every time is really time consuming. So we could create a project template once and initialize projects or it afterwards cookie cutter. This is my cookie cutter template that consists of the two I just
24:23
mentioned and also a GitHub action and some documentation generating tools inside this template. And the only thing you need to do is pip install cookie cutter, then add my URL to cookie my cookie cutter template. It will first
24:42
ask you some question like, what's your project name? And in this example, I type Python table manners. And my template also lets you choose your dependency management tools. And this is how a general rated project looks like. So how to make a template, you'll need to first
25:04
add a cookie cutter that Jason, the key is the value you want to fill in your template. And the value is the default value for that key. And in this example, it's well X user whether which which dependency management tool they
25:23
want to use. And this is a project template of my camp project structure of my template. The upper part is cookie cutter configuration. And the lower part is the template I use to create it. And cookie cutter follow the
25:44
syntax of ginger. This is an example in my template, how I initialize my environment. If the user choose pimping as their dependency management tool, their init task will contain the command pimping install. If they are choosing pointy,
26:02
they'll learn in if the pie will run pointy install. And sometimes you might want to run some check or some operation before or after the project is generated. You can do so by adding hooks, post post or pre generate project
26:23
apply. In this example, if the user does not choose pimping as their dependency management tool, I will remove pimp file for them because pimp file is not needed for other cases. And again, this is my my cookie cutter template. So your
26:44
journey toward a bad manner is now complete. There are other interesting tools that I don't have time to mention today. And you might want to take a look at it. And these are the related talks. I suggest you reading. So does
27:03
anyone has any questions? Yeah, we can also check on this call channel total talk pass and table manners. Yeah, perfect. Well, thank you. Thank you very much. I think we have time for one question. I just don't want to do this to you. So um, let's see. So yes, Gus is asking, do you find
27:24
that committees and slows down your workflow? He finds it hard enough to speed his commits up and not commit everything at once. I value readability more than efficiency, because you can
27:42
surely you can add all the things into into one commit. But if you want to roll, roll back to two certain points, you will be hard to find. And if you make something like refactoring and feature feature and bug fix into one
28:02
commit, and you suddenly find funnel that the bug fix it doesn't actually fix anything you want to roll back to. And after you roll back that commit, your feature will be gone, your refactoring will disappear as well. So I was still recommend you to keep the commit simple. And yeah,
28:23
let's net out. Perfect. Perfect. Well, thank you. Thank you very much. For everyone that is that has questions. I have loads of questions here for you. Unfortunately, we don't have any more time. Yeah, I just made like a block of four of them for you to answer on your channel. And we have,
28:44
yeah, we have a few more. I hope that answered your question because girls also has a few more questions for you. So please just go into this course. The channel is that talk Python table manners. Thank you very much.