Start strong!
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 131 | |
| Author | ||
| Contributors | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/69502 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 202422 / 131
1
10
12
13
16
19
22
33
48
51
54
56
70
71
84
92
93
95
99
107
111
117
123
00:00
Information securitySoftware developerLibrary (computing)Projective planeVideo gameFocus (optics)File formatBitWebsiteRevision controlScripting languageTemplate (C++)Line (geometry)Process (computing)Computer animationLecture/Conference
01:54
Sample (statistics)Data structureDirectory serviceNetwork topologyCorrelation and dependenceProjective planeRight angleSampling (statistics)Single-precision floating-point formatDirectory serviceReading (process)CodeComputer fileInheritance (object-oriented programming)Line (geometry)MereologyCore dumpCASE <Informatik>Dot productComputer animation
03:25
AreaType theoryProjective planeLibrary (computing)CodeSoftware frameworkCASE <Informatik>Computer animation
03:58
Length of stayQuality of serviceoutputInformation securityInheritance (object-oriented programming)Revision controlNumberHash functionPatch (Unix)Software bugEquals signBitRight angleSoftware developerOperator (mathematics)Projective planeComputer animation
06:14
CompilerMereologyComputer fileImplementationExterior algebraComputer animation
06:44
CodeArithmetic meanPersonal identification numberRevision controlProjective planeRight angleComputer fileSoftware developerTouchscreenCompilation albumMultiplication signHash functionDifferent (Kate Ryan album)outputOvalSoftware testingComputer animation
08:50
Curve fittingOrder (biology)LengthLine (geometry)File formatSoftware developerCodeGoodness of fitCodeComputer animation
09:47
AutomationFile formatCodeConsistencyRevision controlMereologyProjective planePauli exclusion principleGroup actionLibrary (computing)Line (geometry)Product (business)Virtual machineMilitary baseConfiguration spaceElectronic program guideError messageType theoryPoint (geometry)Process (computing)File formatMathematicsMultiplication signSynchronizationInheritance (object-oriented programming)Decision theoryTask (computing)Power (physics)QuicksortDifferent (Kate Ryan album)Category of beingComputer animation
13:34
TypprüfungAbstract syntax treeHand fanAutomationMereologyType theoryFocus (optics)PiRight angleComputer fileCASE <Informatik>SynchronizationStandard deviationLibrary (computing)Configuration spaceOrder (biology)Computer animation
15:03
Hash functionAutomationDirectory serviceSpacetimeComputer fileText editorPower (physics)Hooking2 (number)MathematicsFile formatCASE <Informatik>EndomorphismenmonoidSelectivity (electronic)Commitment schemeOcean currentComputer animation
16:52
Computer fileCASE <Informatik>File formatComputer fileMathematicsCommitment schemeQuicksortINTEGRAL
18:03
Software testingQuicksortCodeProjective planeLevel (video gaming)Computer animation
18:27
Software testingSample (statistics)Configuration spaceNetwork topologyDirectory serviceAutomatic differentiationSoftware testingCASE <Informatik>Virtual machineCodeHookingSampling (statistics)Configuration spaceLine (geometry)Speech synthesisProjective planeMessage passingComputer fileReading (process)Standard deviationTheory of relativityLibrary (computing)File formatOcean currentState of matterComputer animation
21:47
Continuous integrationConservation of energyContinuous integrationSoftware testingModule (mathematics)Server (computing)Electric generatorMathematicsBootstrap aggregatingLibrary (computing)CollaborationismSound effectCodeMultiplication signVirtual realityBookmark (World Wide Web)Process (computing)Computer animation
23:22
Process (computing)AutomationGroup actionSoftware testingCAN busPhysical systemProjective planeGroup actionMereologyComputer fileProcess (computing)Greatest elementMultiplication signCASE <Informatik>Computer animation
24:56
Text editorCodeSoftware repositorySoftware testingComputer fileDebuggerProjective planeConfiguration spaceServer (computing)Data managementSet (mathematics)Computer animation
26:06
Network topologySample (statistics)Software testingConfiguration spaceCodeData structureTemplate (C++)Directory serviceCodeLibrary (computing)Template (C++)SoftwareRevision controlProjective planeSoftware testingSampling (statistics)Different (Kate Ryan album)Multiplication signMultiplicationBitComputer fileExpert systemResultantOpen sourceReading (process)Directory serviceComputer animation
28:12
Design of experimentsSet (mathematics)Open setHookingSoftware developerPhysical systemTheory of relativityInstallation artWeb-DesignerFunctional (mathematics)ImplementationLibrary (computing)Configuration spaceFile formatScripting languageMultiplication signAutomatic differentiationDirectory serviceRemote procedure callComputer animationLecture/ConferenceMeeting/Interview
Transcript: English(auto-generated)
00:04
Thank you for coming. I know it's hard to you know pay attention for the last hour before lunch Especially on a Friday, so I appreciate it So yeah today, we're going to be talking about how you can start strong when you're starting a new Python project What are all the different things that you can do to make yours make your life much harder kind of down the line?
00:26
Because the idea is that today we have so many so many different tools that can That can help you with the with the development process, and that's going to be mostly the focus of this talk and Their purpose is to make things easier make things smoother
00:42
Primarily to eliminate kind of the the unnecessary work whether it's discussions with your co-workers on oh How do you ever still format that what tool do you use for that? What version do you use for that? but also make sure that you're effective and Kind of secure in your development that you have things set up so that they help you as opposed to hinder you
01:03
so we're going to be creating like together like a template of a project of an empty project and Putting in all the different tools that we need not only to for the for the Python itself but also stuff things around it We're going to be focusing more on a building a project so something that actually uses Python to to do some things
01:28
We're not that much going to be focusing on creating a library and publishing into pi pi We're going to talk a little bit about it at the end, but most of it will still will still apply But this is more towards. Yeah, I want to I want to create a project. I want to write a website
01:42
I want to write some scripts or something like that for myself or for my team for my company And I want to I want to run it eventually So that's where that's where this is coming from and so let's start with like a super simple setup, right? You I assume that you we're using git for this for this example
02:02
But it will work just as well with other things and we have we just have four files in one directory So we have the sample directory and the under under in it. That's actually our Python code. We created empty and And that's there and Honestly, that's the that's the core of it of obviously But every good project should have also a readme and a license even if you're doing something for yourself
02:29
Spend a few minutes look up a license even if you just say they're all rights reserved or something along those lines Make sure that there is a license to kind of avoid problems down the line It's super easy to set up. There are
02:43
Way more qualified people than me to talk about licensing so I'm gonna just carry on then readme is obviously the most important part because that's the documentation and Every kind of single step that we're going to talk about here today It should end up somewhere in the readme because you should be also documenting about how you work
03:02
What how do you set up the project? How do you manage dependencies and all of those things? and finally obviously dot git ignore That's that's pretty standard so that you don't accidentally commit your PYC file or your virtual end for whatever else you you have there Absolutely, nothing surprising there
03:21
So let's start kind of adding on our tools First we need a way how to kind of install our tools. So that's going to be the first use case on How do we how do we manage that? How do we manage our dependencies? So The dependencies we have multiple different types
03:40
We have the dependency of the project itself of the code the libraries the frameworks that you want to use But also in a similar vein It's the tools that you want to use to actually manage that and obviously that's gonna come first because we haven't yet started writing our code So let's start with that But first a little bit about what what dependencies are usually you have a requirement in your mind like yes
04:06
I want to use fast API and Right now the version of fast API is 0 1 1 1 something else So this is really what you as a developer want to have I want a fast API of that version or something that's compatible with it
04:23
So that's the that's the operator that you're going to use that's essentially pretty much The only one that you need to know that's the tilde equal sign and that mentions a compatible release So it's a release that has the same These two these two numbers zero and one eleven and then it can it can differ on a on a patch releases
04:45
So if there is a new version of fast API released with a with a bug fix or something like that It will it will still match this and it will get it will get applied So this is your requirement. This is the input This is what I want for my project, but that's really not ideal when it comes to deploying things
05:04
There you want to be much more precise Really you want something like this? So this is I want fast API. I want exactly this version one zero one eleven zero and I want one of the packages that matches these these hashes. That's there for security reasons
05:25
So that I know that I'm getting what I ordered when when somebody wants to install install these dependencies and They have a requirements file like this they know that There hasn't been no damage in the shipment and what they actually ordered from from pi pi is actually what was what was delivered
05:46
Super important thing to have and not that complicated, but obviously you don't want to ever do this by hand Because it's not only going to be there for your one requirement But for all of its requirements and so on and so forth
06:02
So there comes our first our first tool where we need to compile this into The into the requirements so from requirements in to requirement dot text So that's where people compile comes in it comes from a package called pip tools Because it does what's on the tin like it does compile the requirements into requirements to txt
06:26
There's an alternative implementation UV which is written rust and it is much faster, so we're going to use that it's compatible And it does exactly what we need to do So how do we manage that part? We need a tool? That's called UV. We need to install it
06:44
Well, we just create another file requirements dash dev txt I Notice a few things here first it already entered in txt. It's a different file I'm not putting it in my requirements dot in because I don't need this to run my code
07:00
I just I need this to manage my code to develop my code So that's why it's in a separate file, and honestly I don't much care about the version because this is something That's always going to be run supervised This is something that I will be installing myself or other developer working on it, and they'll be staring at their screen So I don't need to be super careful and albeit honestly. It's not usually worth the hassle of
07:24
Doing the compilation and all of those different things because then you kind of run into the Chicken-and-the-egg problem like I need a tool to compile my dependencies, but my dependencies need to be compiled and no, so for for
07:40
Requirements dash def I prefer to keep it keep it simple if your project is more is more serious by all means It's always good to do more and pin pin the dependencies even here But that's I think one corner that you can that you can easily cut and so now we have now we have
08:00
UV and requirements at xt we can install it and then just and just then just compile our requirements that In which is just UV pip compile our input. We we shouldn't forget the generate hashes That's what we want. That's how we get the the pretty hashes, and it looks sophisticated and
08:21
And we write it into into the requirement dot txt pretty pretty straightforward So now we have a way how to How to manage the dependencies all we need to do is we need to add it to the requirements dot in Run our pip compile it will generate requirement dot txt and we can install that and we can we can run our code
08:45
We can run tests. We can do all of these different things So might be time to start writing some code, right Well there is a problem If you ask five developers about how what's how a code should look like or what?
09:03
What's considered good code or nice code or something like that you get seven different answers and? And that is that is a problem because not necessarily because any any of their opinions is wrong but I And probably you and I neither don't want to have that discussion. Oh what formatting should we use?
09:25
Should we is it okay if we have like forgetting imports or something like that do they need need to be in alphabetical order? Or should we order them by you know the length of the line or or something like that and those are many problems like that That aren't really that important, but especially some people have a tendency to kind of fixate on that
09:47
So that's why I like to automate these things Obviously just decide decide for once and then just automate it and you never have to have this discussion again That's kind of the the the topic of this of this talk
10:01
I want to automate it so that I don't have to do it, and I don't have to think about it Whether it's me writing alone Or whether it's me writing as part of a wider team Where we would otherwise need to sync on that and and doing doing things like that No, especially when we're starting a new project. It's super easy just decide on one tool and go with it
10:22
So there are different kind of tasks that fall into this category of these discussions So first of them is like linting So you want to link your code which means look for not necessarily errors in the code? But kind of transgressions against the best practices so looking for code smell or something like that
10:46
These are these are the two most popular tools rough and and flake eight I'm gonna be using rough in my examples But honestly it really doesn't matter the point of this is as long as you pick a tool and automate it
11:01
It doesn't really matter that much which tool it is there because there are plenty of reasonable tools that do an amazing job and most of the the other parts come down to personal preference or Strict requirements for example rough for really large code bases. It's much nicer because it's much faster
11:21
and It is more configurable in some ways Less in others etc Generally for your common project when you're when you're starting when you're playing around it doesn't really matter that much So for linting we have a rough check Then we have formatting so not only just check whether the formatting is correct
11:43
But if it isn't actually reformat the code so that it matches but eight and any kind of other style guides that you might have Again, not something that you ever want to do your do yourself and not something that you want to spend your time on When you're talking to your team team members, you don't want to focus on
12:02
Oh, you submitted a pull request, but like could you could you change this to two lines and could you move this over there? Etc. That's not a productive discussion and you've burned valuable time and attention on something that Shouldn't matter as long as it is something that can be understood and resolved by a machine a machine should do it
12:22
that's why you invented them like because so that we can be lazy and focus on the important stuff and Finally, it's kind of a sub Sub problem of the formatting is to sort imports make sure that that it's consistent that it's divided According to the peptide into the three groups, you know the standard library your dependencies and your own your own code
12:46
just for readability and for consistency so that It also helps with resolving merge conflicts and other things It's just a nice thing nice thing to have and as long as you can automate it and it doesn't actually cost you any Brain power to do it. Why not?
13:02
So those are the those are the three tools as you can see it's either flake 8 black and I sort or rough all the way They are mostly compatible. So if you ever decide to switch from one to another you should be you should be absolutely fine There might be some changes during the transition But usually fairly minor and even for like a medium-sized code base. You can you can conflict comfortably do it
13:26
Don't sweat the decision too much just whatever whatever you find go with that and see where that gets you And there is another part of type checking which I'm kind of not gonna focus on because I personally don't do that
13:41
in Python You know your mileage may vary but if you're interested in there There are two main tools that do that Which is my pie and pie right and you can integrate them the same way that you would use the other tools Which is typically Starting with something like pre commit
14:01
So pre commit is a general name for a tool that runs in git Before you commit something. It's like really surprising given the name, right? And it kind of runs some checks and if any of those checks fails, it will tell you. Sorry No, you you have to you have to fix it before before you can before you can commit
14:23
You can also use it for some other things like for example If you have some order generated files or something like that, you can you can you know Generate them using the pre commit so that you you know that you're a master and the and the generated one for example
14:40
Requirements in and requirements of txt are always in sync and that's one of the things that we're going to use it for There is also a tool specifically called pre dash commit Which is a handy little tool for running these standard tools that that we're that we're talking about and there's a huge library out there of Configurations and and tools that you can use so in our case it would look something like this
15:06
We have some built-in built-in hooks like endo file fixer and trailing whitespace. We just make sure that oh, sorry about that That Just make sure that any file that you have won't have any Any trailing whitespace at the end of the line or at the end of the file?
15:24
Just some general bookkeeping. This is again something not super important But it's nice to have it if it doesn't cost you anything any any brainpower. You should have it because otherwise Somebody might open it in an editor and it will reformat the entire file and then you have weird people pull requests
15:41
Where you have unrelated changes, etc. Not super pretty Then we have our our three rough commands the rough format to auto format everything rough itself for the linting and rough select dash I dash fix that's the that's the ordering of the imports and
16:00
Finally, we have a we have the UV With a pre comet. So this will actually make sure to generate a generate the requirements dot txt What you can typically configure for these for these commit hooks which in this case it comes already pre-con Pre-configured is it watches for certain files?
16:21
So for example the requirements hook the last one which can sometimes take like a few seconds or something like that and you don't Want to wait for it when you don't need to it will only run if you're actually committing a change to one of the requirements files so it's it's so it's smart that way and Then all you have to do is run pre commit install which will which will add
16:45
itself into your current git git directory and you can start using it and Using it is pretty straightforward. You just want to commit your awesome changes and things yell at you
17:01
And it's as easy as that In most cases for example the rough format. It will actually change the file So if you're if you're okay with it All you have to do is kind of rerun the command like do another git add git commit and you and you're good to go this but for For other things it might be it might be a little different in this case
17:23
They're just the two to formatting options that that that blew up and my and my pip compile So all I have to do is inspect the changes that it did Add them add them to git and do another commit and I'm done So very very straightforward
17:41
Integrates well with you with your workflow, and you don't have to think about it because you will actually get get poked So that's nice. I'm my mind is now free. I don't have to think about formatting issues or anything like that I can I can sleep safe How cool is that
18:02
Okay So now we kind of sort of dealt with the with the code quality itself or with the code on kind of the Technical level so now we need to start running some tests because ultimately we are here to write some code Which needs to solve a problem, and we need to check whether it solves the problem
18:22
And that would be by creating some creating some tests So I always tell people to start with a test when whenever you create an empty project Add a test in there that that passes or doesn't pass. It doesn't doesn't matter too much But make sure that there is a test to begin with
18:41
the reason for this is the reason kind of why All I'm saying today should be done before you start coding It's always easy to add things if you already have your pre commit installed It's easy to add another hook for example You start adding some YAML files in there for some configurations or other things
19:01
There is a pre commit hook that will that will format the YAML for you So you don't have to you don't have to do it yourself, and it will again be consistent It's much easier to add it to an existing configuration file than to say okay. Oh now. I need to format YAML I need to do that etcetera. No I I'm not gonna do that
19:21
You want to pre-empt any situation where you might be tempted to say oh, no, that's too much work for now That's like a side quest. I I don't want to go there right now. I need I'm focusing on solving my problem so that's why we're doing all of this work up front and one of that is creating a test and even even a fixture as you can see I'm using pytest which is a
19:42
Testing library for for Python it is a Python library. It would go into our requirements dot dash dev so that so that it's so that it can be run there and I even usually create a fixture so you can see this is a fixture for pytest It's kind of like a pre-cant test data. You might you might say so in this case. It just returns to and my test adds
20:08
Confirms that 2 plus 2 does actually equal 4 so super trivial Not really interesting at all, but now I don't have any excuse if I write some code
20:21
I already have a place where I can put the tests for it and hopefully, I will already have the infrastructure in place to run those tests both manually and Automatically so that again. I don't have to think about it In case you haven't noticed that's kind of the theme of the talk like I don't want to think about things I just want machines to do things for me
20:44
So this is this is our current state we've added a bunch of things we've added the requirement We're we've added the test so kind of test is like a standard for pytest where it stores the fixtures and any kind of other Related configurations or anything like that. We have the one test in my test sample, and we have the pre comic config
21:05
Fairly fairly straightforward again remember that all of these things should be always added to the readme If even if it is just one line oh we use pytest for testing To run it just install dev dependencies and run pytest Even just a line like that will definitely help with anybody who's coming into the project or wants to try it out
21:26
And that includes you after two weeks of vacation Because you forget things I forget things that's natural because I don't want to think about things again same thing
21:41
So speaking of I don't want to think about things. I don't want to I don't want to do things do things manually Ideally you want to run tests not only yourself, but also have it be run somewhere else automatically That's something called continuous integration often often shortened to just CI and
22:01
That's something that will run the tests Somewhere on the server whenever you push changes or whether anybody proposes changes to your code and It's important for for these reasons, but also for some that might not be immediately obvious One of my favorite kind of side effects of having continuous integration is you have very you are verifying that your requirements
22:24
Are still valid and that they still make sense? Because let's face it you're not going to recreate your own virtual environment every time you want to do a change or something like that That's work. That's what we're trying to avoid. They're just busy work that we don't want to do But occasionally you might for example forget to add something to the requirements you just install it manually just do pip install an
22:47
Awesome library you use them in your use awesome library in your code you use it in your tests everything is happy locally, but then you push it and Even when you don't have any other collaborators continuous integration can still help you because it will say oh
23:02
I don't I don't have this awesome library that you're talking about like Module not found you need to you need to install it so you know you need to add it to your requirements in Have pre commit generate requirements at txt push it and suddenly you're happy so it verifies your your setup kind of your bootstrapping process
23:24
My example here is with github actions again kind of the most common thing, but any kind of Ci system will will do and again. It's not super complicated and this is a test that a Pipeline that does two tests one is actually run the tests
23:41
That's the top one and the bottom one you can see that. I'm running the pre commit Because I have pre commit I have a thing a tool that checks for a bunch of things It would be a shame not to not to run it in the ci as well And I can actually run it on all the files So not just the files that that have been changed recently in the last commit or in the last push or whatever
24:04
but All the files that I know that everything still fits together It's not definitely something that I want to be running locally because it might take some time And I don't want to stop and wait for it, but once I have something like a ci again It's easy to add stuff add stuff to it
24:23
So and you can see it's it's pretty straightforward You just install your requirement you run your tests in the case of pre commit There is already like a package action that you just use all you need to all it needs is a Python again prepackaged Action to do that so pretty straightforward And there is no reason to not have this as part of your as part of your project from the very beginning
24:47
Because it's there and also notice it doesn't even Reference the name of the project you can be reusing this file very easily one more extra step that you can kind of go is
25:01
Preconfigure editor especially when you're working with a team, and you're kind of using the same tool For example if you're using if you're using vs. Code you can actually add something like this something like this to your to your repo So that would be your your settings or Jason for for vs. Code and this actually tells vs. Code that yes, this is a Python project. I'm using rough for formatting
25:24
I'm using pytest for tests and that way when you when somebody opens this in in vs. Code they can just click run tests, and it will run the tests and that's kind of cool You can also create another file It's called launch the Jason where you can add configuration like how to debug the tests
25:42
Or even like how to run a dev server if you're let's say doing a fast API project or a Django project or something like that So it can tell vs. Code oh yes, you can do Python managed UI run server And if you've run it with your debugger you you can then start start debugging things and because this is common for the project
26:01
You can just push it in there and and have it there shared by everybody else So this is our final result Since the last time we saw it we added the dot github and VS VS code directories, but otherwise, it's it's pretty it's pretty standard and It should do everything that we that we needed to do
26:23
so finally what happens if we want to then take this and publish it to pipi if we want to create this into a Full-blown a full-blown package that we want to distribute so first we need to kind of replace or Augment the requirements of in file with pi project omal I
26:42
There are again much better experts to talk about that or Pi PA which is the the project that kind of maintains the packaging infrastructure They have a sample project specifically for this so that is that is a great resource What's also important not to forget is contributing dot MD if you followed what I've been what I've been saying
27:05
You should have most of those important things in your readme But it's nice sometimes to be a little bit more explicit and add instructions on how to get involved Especially if your code is open source, but even if you're just you're just doing it inside your company
27:21
It's still nice to have that you might even include things like a template for getup issues or pull requests Etc and finally when you're doing this as a library you want to be a little bit more Careful about testing and testing for example multiple different Python versions or testing different versions of your dependencies and libraries
27:44
What I like to do is sometimes just run the the From the pip tools package run the PIP compile with that ish upgrade So it'll try to find the latest version of the soft of the dependency software to make sure that my code still works with them So if there is a if there is a release
28:03
I My my code wouldn't get my code wouldn't get broken and I'm kind of prepared for the future to migrate to the next path And that's pretty much it from my side So if we have if we have any questions about how to get how to get set up so that you don't have to think
28:22
Please ask away. We have a microphone in the middle. Yeah Thank you for the talk. It was really interesting. Just one question about pre commit Is it like a hook or is it like a library that you install it?
28:42
And is it possible to if it's not a hook is it possible to configure it in a git hook? so so it is it is a tool that kind of in and that you install that you So you install it either in your system or you put it in your requirements? Dev because it is a Python tool so you can install with PIP and
29:02
Then you do the the pre commit install and it creates a script in your dot git directory That actually runs runs these hooks. So it's it's it's a function yet supported by git and it's just like ads ads itself in there Okay, so it automatically makes like a pre hook whenever you do git commit then exactly. Yes. Okay. Thank you
29:23
That's a functionality of git itself. So this just like is an implementation of that It's not a question rather a comment on this formatting and limiting because I had learned it a few years ago for a very
29:41
Experienced the lead lead developer slash architect on our team that it saves them a tremendous amount of time if you can share these Configurations within your team and get everybody working in this and that's what most people don't think when Especially when the guy spent more than one sprint You know creating these sharing them and educating the team on how to use because he had to work with remote developers contractors, etc
30:02
Especially when your people are changing in the team, so it's very useful. Thank you Relation is always great. Thank you for your questions and thank you a lot for your talk Konsa Unfortunately, we have no more time for other questions, but please feel free to contact Konsa. We are discord. Thank you again for your talk