We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Automating testing and deployment with Github and Travis

00:00

Formal Metadata

Title
Automating testing and deployment with Github and Travis
Title of Series
Number of Parts
132
Author
License
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
Maintaining an open source project can be a drag. Especially when you have dozens of them. Besides being tedious , making a new release can be a very error prone process. Maybe you forgot to run the full test suite, forgot to update the release version or tag the release on VCS or maybe you bungled up the upload to PyPI. Fortunately there are things you can do to eliminate entire categories of human errors in the release process – by automating them. This talk uses a trivial project as an example to present a release pipeline, from packaging and testing to deployment, that only requires you to push a git tag to your Github repository. Automation then takes care of the rest and you end up with a new release on PyPI (yay!). This talk is mostly useful for open source developers using Github to house their projects, as it heavily leans on Travis, its continuous integration provider . You also need to be at least somewhat familiar with Python packaging and testing in order to follow along. Familiarity with common tools like pytest and tox helps but is not strictly required.
Keywords
35
74
Thumbnail
11:59
View (database)Process (computing)Software maintenanceNumberProjective planeOpen sourceMultiplication signAuthorizationSubsetComputer animation
AutomationError messageSoftware maintenanceOpen sourceRevision controlDistribution (mathematics)Directory serviceComputer configurationElectric currentBuildingLevel (video gaming)Projective planeError messageRepository (publishing)Process (computing)Revision controlComputer programmingInternetworkingQuicksortMereologyWebsiteLevel (video gaming)Multiplication signVector spaceBeta functionNumberMultiplicationConfiguration spacePoint (geometry)BuildingComputer fileDistribution (mathematics)Software testingOpen sourceDefault (computer science)State of matterComputer configurationLoginMathematicsValidity (statistics)Order (biology)Software maintenanceUniform resource locator
Phase transitionSoftware frameworkWeb pageTransport Layer SecurityHacker (term)Virtual machineRepository (publishing)Analog-to-digital converterDisk read-and-write headCodeStatistical hypothesis testingMetadataConfiguration spaceCommon Language InfrastructureInclusion mapCloningNavigationWindowShift operatorMoving averageEmailView (database)Open setComputer fileBit rateComputer-generated imageryThumbnailLaptopPivot elementComputer clusterMagneto-optical driveUser interfaceMenu (computing)Default (computer science)Numbering schemeTemplate (C++)Clique-widthFunction (mathematics)Formal languageCodierung <Programmierung>Term (mathematics)Revision controlPhysical systemGenderLogarithmCodeConfiguration spaceMetadataProjective planePhase transitionBitProcess (computing)Multiplication signAdditionSoftware testingText editorCloningInterpreter (computing)Point (geometry)QuicksortControl flow graphRight angleProduct (business)MereologyAsynchronous Transfer ModeBookmark (World Wide Web)Computer animation
Computer virusStatistical hypothesis testingExpected valueRegulärer Ausdruck <Textverarbeitung>Hand fanCodeComputer fileView (database)WindowNavigationWechselseitige InformationOpen setBookmark (World Wide Web)MetadataSample (statistics)Artificial neural networkGamma functionComputer configurationMaxima and minimaSoftware testingMaxima and minimaRevision controlComputer fileText editorCASE <Informatik>Musical ensembleControl flow graphDirection (geometry)AdditionProduct (business)RootMetadataDirectory serviceComputer configurationStatistical hypothesis testingProjective planePoint (geometry)PiSet (mathematics)Computer animation
Wechselseitige InformationTerm (mathematics)Maxima and minimaView (database)WindowNavigationStatistical hypothesis testingArtificial neural networkControl flowRevision controlCodeComputer fileGamma functionZoom lensConfiguration spaceLocal ringContinuous integrationSuite (music)PasswordEncryptionVery long instruction wordInternet service providerCache (computing)Installation artDistribution (mathematics)StatisticsFormal languageScripting languageRepository (publishing)Order (biology)Software testingMaxima and minimaOpen sourceRevision controlComputer fileConfiguration spaceFree variables and bound variablesProjective planeParameter (computer programming)Suite (music)Point (geometry)AdditionPasswordMereologyMetropolitan area networkCondensationStatistical hypothesis testingRight angleLibrary (computing)Computer animation
View (database)Formal languageWindowNavigationCache (computing)Installation artMathematical optimizationLevel (video gaming)Repository (publishing)Point (geometry)Musical ensembleClient (computing)Declarative programmingComputer animation
View (database)NavigationInstallation artFormal languageWindowCache (computing)PasswordSound effectDemo (music)Electronic mailing listComputer animationSource code
PasswordFormal languageCache (computing)WindowNavigationView (database)GradientHydraulic jumpDistribution (mathematics)EncryptionMenu (computing)AreaLevel (video gaming)CodeMathematical analysisSoftware repositoryServer (computing)Statistical hypothesis testingDuality (mathematics)Analog-to-digital converterRepository (publishing)Wave packetCloningConfidence intervalTransport Layer SecurityRepeating decimalDifferential algebraic equationSimultaneous localization and mappingCellular automatonObject (grammar)Data compressionThread (computing)Total S.A.EmailInstallation artScripting languageInternet service providerRevision controlRegulärer Ausdruck <Textverarbeitung>Process (computing)Configuration spaceLevel (video gaming)Multiplication signFluxGame controllerGraphical user interfaceSoftware testingRight anglePatch (Unix)2 (number)EncryptionCASE <Informatik>Game theoryCoefficient of determinationAreaDifferent (Kate Ryan album)Type theoryWebsiteComputer animation
Transcript: English(auto-generated)
Hello everybody and thanks for coming So today I wanted to talk about Automating very tedious and error-prone things in releasing new PyPI packages
So some background first I've been contributing to open source for 21 years now and Over the years, I've accumulated a number of projects that I'm either the author or maintainer of
And during that time I've learned by I Learned a hard way what can go wrong in the release process? If I can just show you
I'm sure You know people with far more many far more projects that they maintain but Even a subset of this if they are very active can take this toll
so the problem with releasing new stuff is When you do packaging a lot of things can go wrong like for example
You might miss some files in manifest or you might forget to tag your release ingot or All sort of things can go wrong and they are really hard to fix Afterwards but even then
You would really like to automate the process for in order to speed it up because if the maintainer feels that the release process is is troublesome then The maintainer is also far less likely to
to make frequent releases so When I was struggling with these problems, I set out to figure out what what I can do to improve the state of things so I
came up with a process that involves a few select tools and The last piece of the puzzle actually Became available last year. I'll talk about it later but If you have ever released anything you may remember that you need to
At the very minimum You need to run pi Pi Python Setup.py sdist bdist wheel and then twine upload, okay but You also should tag your releases and you should amend your change logs and whatnot
so some of that can be automated and I will show you what I have done to do that. So the process goes as follows
first you make a tag you push it to github or whatever wherever and then github Notifies Travis that there is a new tag and then Travis starts
the build process and Assuming everything goes well Travis then automatically uploads the packages to pi Pi
So, how do we make this happen the first piece of the puzzle is Set up tools SCM What
Right Well, it's good to hear it kind of validates it. So What setup tools SCM does is it's actually two things. First of all It eliminates the need for a manifest.in
If you don't happen to know manifest.in is a file required by setup tools Which tells which files to package into the source distribution? Well with setup tools SCM you don't really need that anymore because it just takes a look at what you have in the repository and
Then just packages the same files in there. So that eliminates one vector of errors And the second one is automatic versioning based on the git tags So This way you will have to add the git tag every time you make a new release
And the version number is only in one location which is in the git repository If for any reason you want to have this dunder version Variable there is a way to do that setup tools SCM has instructions in its documentation for that
The The bonus feature is also that if you make additional commits Then it automatically increments the version which is configurable. I don't like the defaults. I
Configured it to add a post topics like a post 1 post 2 post 3 and so forth There are a lot of configuration options here that you can use to tweak it you are liking Now the second piece of the puzzle
Is a feature of Travis called build stages It was actually launched as far back as last year But nine days ago it went from beta to an official feature so the point here is that
You have the separate stages where all the jobs in a single stage run concurrently and Only when all the jobs have finished will it launch the next stage so What you can do with this is You can make the deploy stage the last one to ensure that all of the actual test jobs
succeed first a Travis has had a pipe here and deploy feature for a long time where After the build it will upload if we able package the project and upload it to pipe here. The trouble was that
If you if you did that without any extra configuration it will just Well, assuming you have multiple Python versions as jobs It will then try to upload this the packages in all the jobs and
Even if you restricted that It could happen that the build fails for some Python version, but not the one where you have to deploy Configured so you would end up with a broken release now build stages fixes that
Also, I like to mention that sometimes you need Experimental builds like I think PIP has a build job on Python nightly You can configure it to
So we did you you cannot be allowed to fail So that it doesn't really really prevent you from releasing But it's nice to know if if it runs on nightly or not All right, now I would like to do a
Sort of practical exercise here assuming internet works and nothing goes wrong
Let's build a trivial project and It is to upload to test pi pi test pi pi if you don't know is Parallel site to pi pi. It's a sort of a sandbox site You don't want to upload these kind of rubbish things on the real pi pi
so in the first phase we will Create a new project on github like the project
I checked beforehand and this this project shouldn't be take this name shouldn't be taken on this pi pi
Yeah This is something that is required I was actually supposed to talk about it
I'm sort of taking all difficulties. So what we are going to do
is After we have a checked out the project on github We will add the code and test
Then we will add the metadata and Packaging configuration that is set up dot pi and set up dot CFG Then we will add configuration for talks and pi test and finally Travis configuration
so the requirements for this is that of course you need a github account obviously and This is something that has changed recently
Travis is now migrating for Travis Is it Travis CI.org to Travis CI.com? It causes some difficulties and that process has changed a little bit But it shouldn't be a problem
In addition to that you will need the Travis command-line tool installed it's a Ruby thing and You need to run Travis login just once I've never had to do it twice, but all in due time
Let's first check out Create and check out the project. Okay, it clones that was the intention now, let me just
Open it in the editor. I think it's a high DPI mode, but we'll make this now
The next phase is the packaging configuration
For now at least set up the PI is always required it may change at some point Depending on who you are asking but
yes it's a requirement but the thing is my personal favorite In a way to configure the metadata is to use setup the CFG which is setup tools feature that was fairly recently introduced and
That is what we're going to do now, but before that we'll create some code
Yes, we should configure Python interpreter for this. It's
Let's use people something like this test for it. Okay a trivial test for this
Okay. Now we have a package and test for it
One side note before we continue it's also a file called pi project Tomo that
you may or may not need but In this case, I just that it was not necessary really It may only be necessary if if you don't have the required
minimum version of setup tools, but other than that, it's not really necessary and Besides it's only work for people 10 onwards. So let's go right those
Set up to pie and set up to CFG now and now we need to use setup tools SCM here
What was the Keyword to look up another project. These are sometimes hard to remember and
You also need To make sure that it's in the setup requirements. So
like this You know, that's pretty much it actually
Most it up the pie files are not this not this short But that's because we are adding the metadata to set up those CFG is it okay?
in addition to the metadata We need some setup tools options if you're wondering about this syntax. It's special. It's a special directive that tells
Set up tools to find any packages in the root directory
Also Since we will be having tests We will need something to run the tests for which we will be using pie test and talks as mentioned before
Some projects add some kind of dev requirements dot txt Or whatever. My way is to add all the Requirements as extras. We will just use
Pie test for now. Okay. I think that's the bare minimum. We still don't have a readme, but it's not
Not necessary for this demonstration, but if you publish anything in a puppy, I make sure you have a readme Now we will need the talks in it
Let's say we want to test on Python 3.6 and 3.7. We want to be running pie test
Last part is just something a
placeholder than where the positional arguments are put if you add any to talks arguments and in order to make talks install the test extra as
Requirement you will want this Okay Let's see how horribly wrong it goes, right?
This Happens because I
Don't really have a git repository here yet So setup tools SCM needs a git repository and at least one commit in order to Determine a version So let's add one. Yeah, we want to add almost all the files here. Okay. Let's see what happens now
I will get to that later, but I'm really new to pipenv actually and I don't feel that it's the the point seems to be with the pipenv is to
pin all the dependencies Which you don't want in the open source library, right? Someone could point out that I'm wrong, but we'll talk about it later For now, we don't need it. All right, we have a failed test
Yes So this was intentional believe me believe it or not
Yeah, because the best way to validate your test suit is to have it fail at first Really so we'll add this and
Now we have passing tests. So it's all good
Now we just have to just amend the first commit
Okay Now we have the bare minimum I think to put it to the test by PI so Before we do that, we need Travis configuration so
For that to happen It's easiest to just copy or Travis configuration from another project. That's what I do. Yeah
There is a thing called Travis a Travis setup pipi It adds some bare minimum configuration to your Travis configuration but
In my opinion it just it just condenses everything and makes it pretty unreadable so What I'm going to do here is Run that very command but only get the encrypted password from there That's the only thing you really need to change in addition to the repository name. So what I'm doing here is
I'm coping the Travis configuration from the wheel project. We don't need code at this point or
notifications Okay, so there are two things that need to be changed here. First is the repository name And
The pipi password for the pipi password I'm going to I'm going to run
the Travis setup thing
I So here Okay, do I put it before or after setup, okay
You know, I rehearsed this before and it worked just fine the comb I Created a new repository today and it works just fine
Yeah, the trouble is that
I I'm sorry for the trouble. This did work earlier today. It's James demo effect
So there is
Encrypted Okay, it worked now So the only thing we yeah
But things are in a flux at Travis, so I'm not really surprised by anything No, I just logged in again
Okay. Now we have a Travis configuration here Let's hope we have time to see the process to the end. Okay. Now, let's see what happens if we push
We need to tag it first. Oh I did Yeah, and then I committed via the GUI Yeah There's one more thing that we need to change
Because we don't we don't want to pollute the actual pi pi. Yes, okay
Now we will add the git tag and push now, let's go see what happens as human works
Are you sure Mm-hmm. Okay, great
right There were a few Item versions that we really didn't need. Oh, right. What happened here is yes
Yeah, we needed the 3.6 and 3.7 Nothing else Yeah, we are running out of time. So We don't have time to wait for the bills to complete so let me just explain how this works
So we have two stages test and deploy to pi pi this last stage is only activated if if there's a tag incoming and This is a regular expression that should match the tag
In this case, it's major dot minor dot patch version Mm-hmm, I'm sorry Alex
Assistant chair. I have the control tie and your time is up and Even though some problem happen with CDL have a sub give some welcome to a second. Thank you