We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Python Table Manners: Cut the Cookie Gracefully

00:00

Formal Metadata

Title
Python Table Manners: Cut the Cookie Gracefully
Subtitle
A Guideline Toward Cleaner Code
Title of Series
Number of Parts
130
Author
License
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
### Goals I expect the audiences to gain knowledge of the tools I mention and the primary usage of them. The tools will cover various aspects of software engineering (e.g., dependencies, testing, security, etc.). Also, I'll purpose how I combine all these tools in my development workflow as a sample for how the audiences can integrate these tools into their workflow. ### Outline for 30 minutes * Dependency Management (4 min) * Testing - Don't let your customer debug for you (4 min) * Style Check and auto-fix (4 min) * Task Management - No more repetitive typing (3 min) * pre-commit - Prevent committing bad code into codebase (3 min) * commitizen-tool - How good commit message can help (4 min) * Security (3 min) * Cookiecutter - Wrap up all the tools (3 min) * Q & A (2 min) ### Outline for 45 minutes * Dependency Management (5 min) * Testing - Don't let your customer debug for you (5 min) * Style Check and auto-fix (5 min) * Task Management - No more repetitive typing (5 min) * pre-commit - Prevent committing bad code into codebase (5 min) * commitizen-tool - How good commit message can help (5 min) * Security (5 min) * Continuous Integration - Assemble all the trivial steps (5 min) * Cookiecutter - Wrap up all the tools (3 min) * Q & A (2 min)
61
Thumbnail
26:38
95
106
HTTP cookieBitMusical ensemblePerfect groupComputer animationMeeting/Interview
SoftwareComputer multitaskingSoftware maintenanceSoftware testingData managementTask (computing)Military operationInformation securityTemplate (C++)HTTP cookieComputer fontSource codeVirtual realityHash functionSynchronizationInstallation artSineLibrary (computing)Standard deviationExecution unitSoftware frameworkPlug-in (computing)MechatronicsPersonal digital assistantConfiguration spaceObject (grammar)DreizehnConsistencyError messagePhysical systemSpacetimeCoding theoryCodeOperator (mathematics)Electronic mailing listInheritance (object-oriented programming)Casting (performing arts)Functional (mathematics)Online helpHash functionSynchronizationRevision controlType theoryVirtual realityTable (information)Software maintenanceProjective planeComputer programmingPattern languageComputer fileSheaf (mathematics)Unit testingSurfaceMultiplication signJava appletArrow of timeMoment (mathematics)Cartesian coordinate systemServer (computing)Object (grammar)Web 2.0Interactive televisionService (economics)Group actionProgrammierstilAsynchronous Transfer ModeExterior algebraSoftware testingConfiguration spacePlug-in (computing)Radical (chemistry)ConsistencyIntegrated development environmentSoftware engineeringFlagString (computer science)Directory serviceLine (geometry)Grass (card game)CASE <Informatik>Cellular automatonProgrammer (hardware)Graph coloringPiMulti-core processorSocial classData managementCodeExecution unitSpacetimeLibrary (computing)Parameter (computer programming)HTTP cookieStandard deviationEmailError messageAdditionReal numberView (database)Visualization (computer graphics)Formal grammarCommitment schemeInstallation art
Configuration spaceExecution unitEmailLogical constantCodePattern languageParameter (computer programming)File formatData typeError messageVirtual machineElectronic mailing listExpected valueExtension (kinesiology)Computer fileComputer configurationLocal GroupLibrary (computing)Standard deviationPauli exclusion principleTask (computing)Event horizonScripting languageInstallation artGastropod shellSoftware developerMilitary operationOperations researchLocal ringType theoryFormal languagePhysical systemAdditionPositional notationLine (geometry)Square numberDirectory serviceString (computer science)Ocean currentMultiplication signDot productLevel (video gaming)Compilation albumFluid staticsCommitment schemeCellular automatonHookingVirtual realityNeuroinformatikRadical (chemistry)TrailComa BerenicesDefault (computer science)Buffer solutionComputer fileComputer configurationField (computer science)Software repositoryOperator (mathematics)Local ringSoftware testingSoftware developerScripting languageGastropod shellType theoryComplete metric spaceTraffic reportingStatisticsRun time (program lifecycle phase)Projective planeUnit testingProgrammierstilGroup actionConfiguration spaceLibrary (computing)Electronic mailing listFunctional (mathematics)Revision controlComputer programmingRootParameter (computer programming)File formatSpacetimeMaxima and minimaWeb pageExtension (kinesiology)Virtual machineCartesian coordinate systemOrder (biology)Arrow of timeTask (computing)CodeData managementZeno of EleaIntegerPi2 (number)Mobile appNamespaceLink (knot theory)Demo (music)Content (media)Patch (Unix)Automatic differentiationBooting
Software repositoryConfiguration spaceTrailRevision controlMenu (computing)Correlation and dependenceSpecial unitary groupCodeMaizeNormed vector spaceCloningDivisorData typeDenial-of-service attackIntegrated development environmentVulnerability (computing)FreewareInheritance (object-oriented programming)Template (C++)Data structureComputer clusterCASE <Informatik>TelecommunicationIntegrated development environmentLoginProjective planeLeakMathematicsFlow separationSoftware bugAnalytic setSingle-precision floating-point formatComputer programmingMathematical analysisVulnerability (computing)Template (C++)Configuration spaceLibrary (computing)Revision controlDatabaseMessage passingDifferent (Kate Ryan album)Data structureSpring (hydrology)Coma BerenicesSpeech synthesisComputer filePhysical systemType theoryInstallation artSpacetimeRepository (publishing)Line (geometry)User interfaceMultiplication signHTTP cookieData managementTask (computing)MereologyDefault (computer science)Information securityCodeKey (cryptography)Functional (mathematics)Uniform resource locatorGroup actionTable (information)Electric generatorRule of inferenceSheaf (mathematics)HookingException handlingSoftware testingGoodness of fitPatch (Unix)Commitment schemeVideo gameStatisticsOpticsPiMedian
Software testingMessage passingTotal S.A.System callTable (information)Moving averageCommitment schemeMultiplication signCode refactoringPerfect groupBlock (periodic table)Software bugStructural loadWeightPoint (geometry)Computer animationMeeting/Interview
Transcript: English(auto-generated)
Perfect. So it's a pleasure to have you here. And you're going to talk to us a little bit about cookie cutter science as well, right? Yeah, you're definitely right. Cool. I am personally interested in this talk as well. Because I really like Kedro
and Kedro is like one step after the cookie cutter data science. So it's inspired on, but we can have a chat about this later on. So yeah, the floor is yours. Take it away. Okay, then welcome to my talk Python Table Manners and Color Cookie Grassfully. Yeah,
I'm Weili. I'm a software engineer at Rockton Slides and also a volunteer at PyCon Taiwan. As you can see, this is a teacher for help from PyCon Taiwan. Also, I am a maintainer of commitment tools, which is our tool I mentioned in this talk. Today, I will illustrate how
to first clean out your table before you get to your dinner. And we'll ensure you that you'll put the correct tableware on your table, then we'll learn how to use this tableware elegantly. Because the step may be too trivial for you, you'll need some mnemonic press. And if you're asking others, like to get to help you, you should
say please, in such an occasion, you'll like to speak formally. And when we have our knife to cut a cookie, we'll ensure our own safety. And the last step we'll see first the cookie. Starting from dependency management, this might be how we
used to start a Python project, we create a virtual environment, and then we activated. And after that, we'll free some packages into requirements.txt. But sometimes we just forget to activate the virtual environment, or we forget to add a package into requirements.txt. So we can use tools like Pippin, because Pippin can
management, manage your virtual environment and packaging at the same time. So you no longer need to manually sync up your virtual environment and your requirements.txt. It's owner also generate hashes from the package choose Pippin download from type
PI. So it's can ensure you that you can get the same packages next time you installed from it. You can initially visual environments through Pippin install. Then this is how an empty PIP file looks like. Pippin views PIP file and PIP file
.loc to manage dependencies as an alternative to requirements.txt. And the API is pretty much the same as PIP. So you just need to type Pippin install, which package is equal to which version. If you add a request to your virtual
environment, it will update a PIP file like this. And it will add this, this section into your PIP file .loc. This hashes is generated from the code Pippin download from PI PI at the last moment. Even if the next time you download it,
the code is changed, but the version is still 2.22.0. Pippin will rise and error so you're guaranteed to have the exact same package next time. But sometimes you just don't need everything in your production environment. So you can install the packages into your development
environment only through adding a dash dash def argument. And it will appear in the def package section in your PIP file. And because we already set our virtual environment and manage our dependency at the same time, we need to run our Python program inside our virtual environment. You can do so
by Pippin run Python your program or something like Pippin run Django management the PI to start the web server. But some people might say that Pippin does not update that frequently or it's just updated two months ago, and the back of it is really slow. And it does not sync up
with install request instead of the PI. Maybe you could try poetry. The concept of poetry and Pippin are alike. So I'll list the command here as a reference. For releasing a package, I will recommend you using poetry because you don't have to manually update dependencies on both
PIP file and the other PI, poetry will do that for you. But for Python application, I'll say both poetry and Pippin work for me. Testing Python comes with standard unit test unit test framework in its
standard library. But it's borrow the concept from J unit in Java. So today I want to introduce pytest. Why should we use pytest? Because pytest is considered to be more Pythonic, and it's compatible with the old unit test style. And in unit test, you will need to use a
assert function like assert equal, assert true or assert false and extra etc. But in pytest, you just need to memorize asserts. And the other side, you can use the same syntax as the normal Python you use. And pytest provides better test discovery, advanced
features, and it also comes with plenty of plugins. This is how we pytest. Actually, after we install our virtual environment, we should always install our packages into our virtual environment and run our Python program inside the virtual environment. This is how a unit test unit test style test
looks like. First, we use the setup to prepare all the data needed in our test cases. But because unit test is borrow the concept from J unit, it's set up function is camel test, which consider which is considered to be non Pythonic, and we'll
inherit unit test test test class. And as I mentioned previously, we'll need to memorize the assert that assert functions. And in pytest, we use fixtures to prepare individual data for individual test cases. So we no longer need the
setup function. Also, we we don't need to inherit a best class. And we don't even need a class to run pytest. And the assert function become much easier to memorize because you only need to use assert and the same test afterward is just
the same as how you use Python daily. And this is my configuration for pytest. I use pytest dot init to configure pytest because style dot config is not recommended for pytest configuration nowadays. And after pytest 6.0 is released,
you can even configure pytest through pyproject.toml. These are the plugins I use in almost every of my Python projects. You will use pytest mark for replace the object letter how to test like if you your program interact with AWS
GCP or others, third party services, you don't want to actually interact with them because it's real cost you money. So you will want to use a fact object to to be testing your unit tests. And pytest coverage can show you
which portion of your program is not covered by your unit tests. And pytest dot xdist can accelerate your test by distributing your test to multi core coding style. As Python programmer, we not only want to write correct code, we
also want to write elegant code, we can do so by flake. Flake is a tool that can enforce style consistency across your Python project. It can also check possible error before you actually run your program, and also eliminate the bad coding styles. In this example, I
redefined the OS library as a string, which could be a possible error because after this line, you can no longer use methods like OS dot get current working directory because OS now is a string. And I add an additional space
here, which is considered to be a bad coding style. After running flake, it will tell you where are the errors and best best mouse. This is my configuration for flake. I use that on the config. In this section, I will introduce
you a lot of tool that relates to coding style. So by following this configuration, all these tools will not conflict with each other. Piling, the functionality of piling is pretty much the same as flake, but it can generate more detailed reports. This is a, we use the
same code and run pilings. This is the report, it's just me. And if you run pilings with dash argument, it can generate a even more detailed report, which you can compare with your previous piling run to see where your coding style is improved in between the two
different check. And I use pyproject internal for configuration for configuring pytest. I used to use pilings RC for configure pytest. I follow that the default pilings RC contains too many default values, which
is distracting when and Maggie make me hard to find the thing I really want to configure. So I think I've seen pyproject eternal or you can use that config to configure pilings, which might be a better solution. My pie in Tyson communities type annotation is now
encouraged. So we now have tools like my pie to do static static type checking. And by doing so, so you can avoid possible runtime error because my pie can run compile time type checking. And by doing type annotation,
it's can enhance your readability. Now, it's not only work as like dark string, it's machine checkable documentation. In this example, values is annotated as a list of stir, but we passing a list of integer into, which will be a runtime error. But tools like
second and pilot won't, won't, won't you about this error, but my pie will, my pie will tell you that you should actually passing a list of stir instead of list of integer. The first argument indicate that you want
to check all the file with the extension. And the second one will ignore the arrow. That's your third party library is not type a note added. Because what we actually care about is where our code is type a note added. And this is how I configure my pipe through
style that config. Let's do nice. The configuration, the other arguments I showed in the previous patch. So after this, you no longer need to type arguments. We can even take one step forward by fixing the style automatically to black. Using black is really easy. You
just need to run that. And it will fix all the styles under your current work directory. This is how that reformat the red one is the code before black actually fix it. Because backslash is not recommended in our day that will use a square to do the black line. And it will
fix the notation and also add an additional space between command and its command and its content. Why should we use black? Because the black code style is not configurable. You cannot tell that how to how to
format or I don't like this portion of black style. No, you cannot, you need to establish the black code style for you. For your whole file, you can not even get it temporary ignore mark to format only a portion of your
code, you need to format the whole file, and which leads to no more argument about which style is better. So you can focus on what really matters. The value feature you want to deliver to your customers. And there should be one and preferably only one obvious way
to do it. It's from the demo Python. And this is my configuration for that. This is not for for black formatting, it's just to tell that which file is you include, and which file is should exclude. I thought in
this, in this Python file, I randomly import some libraries. But according to paper, we should start our libraries in the following order, we should first have standard libraries. Second, third party library and serve our local application or library imports. And you
should add a blank line between each group. After running, I thought it will group this import for you. In addition to group is it will start the libraries alphabetically. So the next time you want to buy, which which library is imported, it will be much easier to find
than randomly sorted. And this is my configuration for I thought this is all the coding style related tools I use for my Python project. And this outcome then for formatting and linking. But it's just way too many
comments. So we need some tools like pipe invoke for task management. It's like a max file, but it's written in Python. I'll demonstrate how we use invoke in report generator command line in practice. This is this
is how we install our analyzer and run unit test before we use invoke will need to memorize this long comment. But after invoke the comment become much shorter. But you might say, even even if they are much shorter, now we still
need to memorize that. But no, you don't. Actually, you the thing only thing you need to memorize is involved dash l, it will list all the comment you implement. So how can we implement this comment. So test that by your app this file to the boot of your Python project, and
then move the command to the tech. And best of all is can ensure that your Python program is run inside your virtual environment by adding virtual environment prefix in your test apply. After you introduce a bunch of tests, you
might want to modularize them through the concept of namespace. So as you can see now, the comment becomes invoke build dot develop. But with your comment become even longer now. So now we need auto completion, invoke, invoke
comes with a completion script, you can generate the script for each of your shells through this comment. And after that, you can type in both built that tab, and it will show you all the options you can choose from. So why not just use Mac file? Well, because we are Python developers, and some
tasks might not be easy to handle through shell script. And just good in differential command might be different. And in Python book, you can combine the power of Python and shell script. It's a best of both. Actually. And because
people might forget to do a check, even after we made a check much easier, so we can ask it to do a check through pre commit. So how do we commit do a check for us? For coming can run some command before we do any get operation
like it's push and get commit. We'll first need to tell pre commit, what command we were to run. And you fix example, I first use a rebel local, which means it will run local command on your computer. And the first hook is style
reformat. I will check it at the stage commit, and I will run invoke style reformat at this stage. The second one is style check. I will do style check when when I do get push. There are also some existing hoops that is commonly used. So
becoming has a repository becoming hooks. For example, I have introduced an optics and file fixers and traveling widespread, which will remove the traveling widespread in your files except my phone files. And popular project
like that I thought and you won't like it has their own hooks on GitHub as well. So after we configure it, we need to install it into your local local repository. So pre commit install, because I mentioned I will use stat push and commit. So I install hook type pre copy and pre push in
this example. After that, you do get commit, and you will run the end of life fixer, trim the white space, and do the style reformat. And when you want to push your code to your remote repository, it will do a first two and then we'll do
style check without style reformat. Speaking of it, we might want to cultivate a git commit convention. If you're like this guy, and you will see it look like this. And it will make it's really hard to find the right version to roll back
to when you your system goes down. Because all the commits update, you cannot distinguish them. So commitment tools is here to help by using the commit, which is a comment from committed said, you can you'll get a user and user interface
that will first ask you which type of change is this one. And it's also gives the user a hint that you should not make a lot of different different changes into a single comment. For example, you should not add bug based feature and refactoring into a single comment with which will
make it really hard to review. And then it will ask you about which scope is it the subject of it, and whether this comment is a breaking change. And you might want to add some some more details to your to your comments. And then you
might want to reference to your GitHub issue, GitHub issue or don't you rotate it. This is a comment we just generate your previous patch. And if you're continuously using commencement, you'll see this kind of good luck, which is much
readable than update. Committances also comes with some advanced features, it can prevent you from not using commencement because people still forget to use commencement and they use it to come in with update. They sometimes and because the rule we just use is conventional commit, we
can use customizable commit rules. And because our coming messages standardize, we can also bank our project version and generate change your own. So come in to send it through the functionality of committee set. And best of
all, I'll hold a spring tomorrow and Sunday. So that spring, join us on virtual run 10. Security issues, you might have seen this kind of warning on GitHub, it tell you that some of your libraries might have security issues
that's usually upgrade your your library to certain version. You can do so locally through safety. You can check the vulnerability by running safety check. In this example, it tells me that pycrypto 2.6 2.1 might be
dangerous. So I need to update to a higher version. But if you are using pipping, you can just run pipping check. Safety will search your vulnerability in the CVS database for for the known vulnerabilities. Embedded bandit can do static analysis
on to check common security issue in your Python program. And in this example bandit tells me that in this portion of my code, I have a median severity security issue and bandit this is a how confident bandit think this is
actually an issue. And this is bandit will also tell you how why this might be a problem and how you could fix it. But not all the warning should be fixed. In this example, it tells me that I should not use a cert. It's
because a cert might be ignoring some Python configuration. So if you are using a cert to identify your user in your logging system, you might be a potential vulnerability. But we still need to use a cert in our test cases, right? So you should add a test into your escrow section into your bandit configuration. But if you are
the whole file, just some section or some lines in your code, you can add no secure after the end of the line of code you want to ignore the warning. So cool. Let's talk about cookies. Project template, you might want to
use all these manners in your all your pricing projects afterward, but configure it every time is really time consuming. So we could create a project template once and initialize projects or it afterwards cookie cutter. This is my cookie cutter template that consists of the two I just
mentioned and also a GitHub action and some documentation generating tools inside this template. And the only thing you need to do is pip install cookie cutter, then add my URL to cookie my cookie cutter template. It will first
ask you some question like, what's your project name? And in this example, I type Python table manners. And my template also lets you choose your dependency management tools. And this is how a general rated project looks like. So how to make a template, you'll need to first
add a cookie cutter that Jason, the key is the value you want to fill in your template. And the value is the default value for that key. And in this example, it's well X user whether which which dependency management tool they
want to use. And this is a project template of my camp project structure of my template. The upper part is cookie cutter configuration. And the lower part is the template I use to create it. And cookie cutter follow the
syntax of ginger. This is an example in my template, how I initialize my environment. If the user choose pimping as their dependency management tool, their init task will contain the command pimping install. If they are choosing pointy,
they'll learn in if the pie will run pointy install. And sometimes you might want to run some check or some operation before or after the project is generated. You can do so by adding hooks, post post or pre generate project
apply. In this example, if the user does not choose pimping as their dependency management tool, I will remove pimp file for them because pimp file is not needed for other cases. And again, this is my my cookie cutter template. So your
journey toward a bad manner is now complete. There are other interesting tools that I don't have time to mention today. And you might want to take a look at it. And these are the related talks. I suggest you reading. So does
anyone has any questions? Yeah, we can also check on this call channel total talk pass and table manners. Yeah, perfect. Well, thank you. Thank you very much. I think we have time for one question. I just don't want to do this to you. So um, let's see. So yes, Gus is asking, do you find
that committees and slows down your workflow? He finds it hard enough to speed his commits up and not commit everything at once. I value readability more than efficiency, because you can
surely you can add all the things into into one commit. But if you want to roll, roll back to two certain points, you will be hard to find. And if you make something like refactoring and feature feature and bug fix into one
commit, and you suddenly find funnel that the bug fix it doesn't actually fix anything you want to roll back to. And after you roll back that commit, your feature will be gone, your refactoring will disappear as well. So I was still recommend you to keep the commit simple. And yeah,
let's net out. Perfect. Perfect. Well, thank you. Thank you very much. For everyone that is that has questions. I have loads of questions here for you. Unfortunately, we don't have any more time. Yeah, I just made like a block of four of them for you to answer on your channel. And we have,
yeah, we have a few more. I hope that answered your question because girls also has a few more questions for you. So please just go into this course. The channel is that talk Python table manners. Thank you very much.