Automating testing and deployment with Github and Travis
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 132 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/44996 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
EuroPython 20189 / 132
2
3
7
8
10
14
15
19
22
27
29
30
31
34
35
41
44
54
55
56
58
59
61
66
74
77
78
80
81
85
87
91
93
96
98
103
104
105
109
110
111
113
115
116
118
120
121
122
123
125
127
128
129
130
131
132
00:00
View (database)Process (computing)Software maintenanceNumberProjective planeOpen sourceMultiplication signAuthorizationSubsetComputer animation
01:24
AutomationError messageSoftware maintenanceOpen sourceRevision controlDistribution (mathematics)Directory serviceComputer configurationElectric currentBuildingLevel (video gaming)Projective planeError messageRepository (publishing)Process (computing)Revision controlComputer programmingInternetworkingQuicksortMereologyWebsiteLevel (video gaming)Multiplication signVector spaceBeta functionNumberMultiplicationConfiguration spacePoint (geometry)BuildingComputer fileDistribution (mathematics)Software testingOpen sourceDefault (computer science)State of matterComputer configurationLoginMathematicsValidity (statistics)Order (biology)Software maintenanceUniform resource locator
09:33
Phase transitionSoftware frameworkWeb pageTransport Layer SecurityHacker (term)Virtual machineRepository (publishing)Analog-to-digital converterDisk read-and-write headCodeStatistical hypothesis testingMetadataConfiguration spaceCommon Language InfrastructureInclusion mapCloningNavigationWindowShift operatorMoving averageEmailView (database)Open setComputer fileBit rateComputer-generated imageryThumbnailLaptopPivot elementComputer clusterMagneto-optical driveUser interfaceMenu (computing)Default (computer science)Numbering schemeTemplate (C++)Clique-widthFunction (mathematics)Formal languageCodierung <Programmierung>Term (mathematics)Revision controlPhysical systemGenderLogarithmCodeConfiguration spaceMetadataProjective planePhase transitionBitProcess (computing)Multiplication signAdditionSoftware testingText editorCloningInterpreter (computing)Point (geometry)QuicksortControl flow graphRight angleProduct (business)MereologyAsynchronous Transfer ModeBookmark (World Wide Web)Computer animation
17:41
Computer virusStatistical hypothesis testingExpected valueRegulärer Ausdruck <Textverarbeitung>Hand fanCodeComputer fileView (database)WindowNavigationWechselseitige InformationOpen setBookmark (World Wide Web)MetadataSample (statistics)Artificial neural networkGamma functionComputer configurationMaxima and minimaSoftware testingMaxima and minimaRevision controlComputer fileText editorCASE <Informatik>Musical ensembleControl flow graphDirection (geometry)AdditionProduct (business)RootMetadataDirectory serviceComputer configurationStatistical hypothesis testingProjective planePoint (geometry)PiSet (mathematics)Computer animation
25:49
Wechselseitige InformationTerm (mathematics)Maxima and minimaView (database)WindowNavigationStatistical hypothesis testingArtificial neural networkControl flowRevision controlCodeComputer fileGamma functionZoom lensConfiguration spaceLocal ringContinuous integrationSuite (music)PasswordEncryptionVery long instruction wordInternet service providerCache (computing)Installation artDistribution (mathematics)StatisticsFormal languageScripting languageRepository (publishing)Order (biology)Software testingMaxima and minimaOpen sourceRevision controlComputer fileConfiguration spaceFree variables and bound variablesProjective planeParameter (computer programming)Suite (music)Point (geometry)AdditionPasswordMereologyMetropolitan area networkCondensationStatistical hypothesis testingRight angleLibrary (computing)Computer animation
33:58
View (database)Formal languageWindowNavigationCache (computing)Installation artMathematical optimizationLevel (video gaming)Repository (publishing)Point (geometry)Musical ensembleClient (computing)Declarative programmingComputer animation
35:54
View (database)NavigationInstallation artFormal languageWindowCache (computing)PasswordSound effectDemo (music)Electronic mailing listComputer animationSource code
36:39
PasswordFormal languageCache (computing)WindowNavigationView (database)GradientHydraulic jumpDistribution (mathematics)EncryptionMenu (computing)AreaLevel (video gaming)CodeMathematical analysisSoftware repositoryServer (computing)Statistical hypothesis testingDuality (mathematics)Analog-to-digital converterRepository (publishing)Wave packetCloningConfidence intervalTransport Layer SecurityRepeating decimalDifferential algebraic equationSimultaneous localization and mappingCellular automatonObject (grammar)Data compressionThread (computing)Total S.A.EmailInstallation artScripting languageInternet service providerRevision controlRegulärer Ausdruck <Textverarbeitung>Process (computing)Configuration spaceLevel (video gaming)Multiplication signFluxGame controllerGraphical user interfaceSoftware testingRight anglePatch (Unix)2 (number)EncryptionCASE <Informatik>Game theoryCoefficient of determinationAreaDifferent (Kate Ryan album)Type theoryWebsiteComputer animation
Transcript: English(auto-generated)
00:04
Hello everybody and thanks for coming So today I wanted to talk about Automating very tedious and error-prone things in releasing new PyPI packages
00:23
So some background first I've been contributing to open source for 21 years now and Over the years, I've accumulated a number of projects that I'm either the author or maintainer of
00:44
And during that time I've learned by I Learned a hard way what can go wrong in the release process? If I can just show you
01:04
I'm sure You know people with far more many far more projects that they maintain but Even a subset of this if they are very active can take this toll
01:29
so the problem with releasing new stuff is When you do packaging a lot of things can go wrong like for example
01:41
You might miss some files in manifest or you might forget to tag your release ingot or All sort of things can go wrong and they are really hard to fix Afterwards but even then
02:03
You would really like to automate the process for in order to speed it up because if the maintainer feels that the release process is is troublesome then The maintainer is also far less likely to
02:22
to make frequent releases so When I was struggling with these problems, I set out to figure out what what I can do to improve the state of things so I
02:40
came up with a process that involves a few select tools and The last piece of the puzzle actually Became available last year. I'll talk about it later but If you have ever released anything you may remember that you need to
03:02
At the very minimum You need to run pi Pi Python Setup.py sdist bdist wheel and then twine upload, okay but You also should tag your releases and you should amend your change logs and whatnot
03:26
so some of that can be automated and I will show you what I have done to do that. So the process goes as follows
03:44
first you make a tag you push it to github or whatever wherever and then github Notifies Travis that there is a new tag and then Travis starts
04:08
the build process and Assuming everything goes well Travis then automatically uploads the packages to pi Pi
04:21
So, how do we make this happen the first piece of the puzzle is Set up tools SCM What
04:42
Right Well, it's good to hear it kind of validates it. So What setup tools SCM does is it's actually two things. First of all It eliminates the need for a manifest.in
05:04
If you don't happen to know manifest.in is a file required by setup tools Which tells which files to package into the source distribution? Well with setup tools SCM you don't really need that anymore because it just takes a look at what you have in the repository and
05:23
Then just packages the same files in there. So that eliminates one vector of errors And the second one is automatic versioning based on the git tags So This way you will have to add the git tag every time you make a new release
05:47
And the version number is only in one location which is in the git repository If for any reason you want to have this dunder version Variable there is a way to do that setup tools SCM has instructions in its documentation for that
06:09
The The bonus feature is also that if you make additional commits Then it automatically increments the version which is configurable. I don't like the defaults. I
06:23
Configured it to add a post topics like a post 1 post 2 post 3 and so forth There are a lot of configuration options here that you can use to tweak it you are liking Now the second piece of the puzzle
06:41
Is a feature of Travis called build stages It was actually launched as far back as last year But nine days ago it went from beta to an official feature so the point here is that
07:00
You have the separate stages where all the jobs in a single stage run concurrently and Only when all the jobs have finished will it launch the next stage so What you can do with this is You can make the deploy stage the last one to ensure that all of the actual test jobs
07:27
succeed first a Travis has had a pipe here and deploy feature for a long time where After the build it will upload if we able package the project and upload it to pipe here. The trouble was that
07:45
If you if you did that without any extra configuration it will just Well, assuming you have multiple Python versions as jobs It will then try to upload this the packages in all the jobs and
08:01
Even if you restricted that It could happen that the build fails for some Python version, but not the one where you have to deploy Configured so you would end up with a broken release now build stages fixes that
08:23
Also, I like to mention that sometimes you need Experimental builds like I think PIP has a build job on Python nightly You can configure it to
08:40
So we did you you cannot be allowed to fail So that it doesn't really really prevent you from releasing But it's nice to know if if it runs on nightly or not All right, now I would like to do a
09:13
Sort of practical exercise here assuming internet works and nothing goes wrong
09:22
Let's build a trivial project and It is to upload to test pi pi test pi pi if you don't know is Parallel site to pi pi. It's a sort of a sandbox site You don't want to upload these kind of rubbish things on the real pi pi
09:43
so in the first phase we will Create a new project on github like the project
10:22
I checked beforehand and this this project shouldn't be take this name shouldn't be taken on this pi pi
10:43
Yeah This is something that is required I was actually supposed to talk about it
11:19
I'm sort of taking all difficulties. So what we are going to do
11:28
is After we have a checked out the project on github We will add the code and test
11:47
Then we will add the metadata and Packaging configuration that is set up dot pi and set up dot CFG Then we will add configuration for talks and pi test and finally Travis configuration
12:12
so the requirements for this is that of course you need a github account obviously and This is something that has changed recently
12:23
Travis is now migrating for Travis Is it Travis CI.org to Travis CI.com? It causes some difficulties and that process has changed a little bit But it shouldn't be a problem
12:43
In addition to that you will need the Travis command-line tool installed it's a Ruby thing and You need to run Travis login just once I've never had to do it twice, but all in due time
13:05
Let's first check out Create and check out the project. Okay, it clones that was the intention now, let me just
14:17
Open it in the editor. I think it's a high DPI mode, but we'll make this now
14:56
The next phase is the packaging configuration
15:08
For now at least set up the PI is always required it may change at some point Depending on who you are asking but
15:22
yes it's a requirement but the thing is my personal favorite In a way to configure the metadata is to use setup the CFG which is setup tools feature that was fairly recently introduced and
15:44
That is what we're going to do now, but before that we'll create some code
16:02
Yes, we should configure Python interpreter for this. It's
17:02
Let's use people something like this test for it. Okay a trivial test for this
19:19
Okay. Now we have a package and test for it
19:34
One side note before we continue it's also a file called pi project Tomo that
19:44
you may or may not need but In this case, I just that it was not necessary really It may only be necessary if if you don't have the required
20:00
minimum version of setup tools, but other than that, it's not really necessary and Besides it's only work for people 10 onwards. So let's go right those
20:21
Set up to pie and set up to CFG now and now we need to use setup tools SCM here
21:02
What was the Keyword to look up another project. These are sometimes hard to remember and
21:56
You also need To make sure that it's in the setup requirements. So
22:17
like this You know, that's pretty much it actually
22:29
Most it up the pie files are not this not this short But that's because we are adding the metadata to set up those CFG is it okay?
23:37
in addition to the metadata We need some setup tools options if you're wondering about this syntax. It's special. It's a special directive that tells
23:56
Set up tools to find any packages in the root directory
24:07
Also Since we will be having tests We will need something to run the tests for which we will be using pie test and talks as mentioned before
24:20
Some projects add some kind of dev requirements dot txt Or whatever. My way is to add all the Requirements as extras. We will just use
24:55
Pie test for now. Okay. I think that's the bare minimum. We still don't have a readme, but it's not
25:06
Not necessary for this demonstration, but if you publish anything in a puppy, I make sure you have a readme Now we will need the talks in it
25:36
Let's say we want to test on Python 3.6 and 3.7. We want to be running pie test
25:59
Last part is just something a
26:03
placeholder than where the positional arguments are put if you add any to talks arguments and in order to make talks install the test extra as
26:21
Requirement you will want this Okay Let's see how horribly wrong it goes, right?
27:18
This Happens because I
27:22
Don't really have a git repository here yet So setup tools SCM needs a git repository and at least one commit in order to Determine a version So let's add one. Yeah, we want to add almost all the files here. Okay. Let's see what happens now
28:48
I will get to that later, but I'm really new to pipenv actually and I don't feel that it's the the point seems to be with the pipenv is to
29:06
pin all the dependencies Which you don't want in the open source library, right? Someone could point out that I'm wrong, but we'll talk about it later For now, we don't need it. All right, we have a failed test
29:35
Yes So this was intentional believe me believe it or not
29:44
Yeah, because the best way to validate your test suit is to have it fail at first Really so we'll add this and
30:34
Now we have passing tests. So it's all good
30:41
Now we just have to just amend the first commit
31:03
Okay Now we have the bare minimum I think to put it to the test by PI so Before we do that, we need Travis configuration so
31:37
For that to happen It's easiest to just copy or Travis configuration from another project. That's what I do. Yeah
31:49
There is a thing called Travis a Travis setup pipi It adds some bare minimum configuration to your Travis configuration but
32:06
In my opinion it just it just condenses everything and makes it pretty unreadable so What I'm going to do here is Run that very command but only get the encrypted password from there That's the only thing you really need to change in addition to the repository name. So what I'm doing here is
32:56
I'm coping the Travis configuration from the wheel project. We don't need code at this point or
33:22
notifications Okay, so there are two things that need to be changed here. First is the repository name And
33:50
The pipi password for the pipi password I'm going to I'm going to run
34:07
the Travis setup thing
35:01
I So here Okay, do I put it before or after setup, okay
35:22
You know, I rehearsed this before and it worked just fine the comb I Created a new repository today and it works just fine
35:50
Yeah, the trouble is that
36:04
I I'm sorry for the trouble. This did work earlier today. It's James demo effect
37:16
So there is
37:28
Encrypted Okay, it worked now So the only thing we yeah
37:41
But things are in a flux at Travis, so I'm not really surprised by anything No, I just logged in again
38:00
Okay. Now we have a Travis configuration here Let's hope we have time to see the process to the end. Okay. Now, let's see what happens if we push
38:42
We need to tag it first. Oh I did Yeah, and then I committed via the GUI Yeah There's one more thing that we need to change
39:04
Because we don't we don't want to pollute the actual pi pi. Yes, okay
39:46
Now we will add the git tag and push now, let's go see what happens as human works
41:08
Are you sure Mm-hmm. Okay, great
41:50
right There were a few Item versions that we really didn't need. Oh, right. What happened here is yes
43:01
Yeah, we needed the 3.6 and 3.7 Nothing else Yeah, we are running out of time. So We don't have time to wait for the bills to complete so let me just explain how this works
43:25
So we have two stages test and deploy to pi pi this last stage is only activated if if there's a tag incoming and This is a regular expression that should match the tag
43:41
In this case, it's major dot minor dot patch version Mm-hmm, I'm sorry Alex
44:00
Assistant chair. I have the control tie and your time is up and Even though some problem happen with CDL have a sub give some welcome to a second. Thank you