Continuous Deployment for webapps based on Django
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Serientitel | ||
| Teil | 126 | |
| Anzahl der Teile | 173 | |
| Autor | ||
| Lizenz | CC-Namensnennung - keine kommerzielle Nutzung - Weitergabe unter gleichen Bedingungen 3.0 Unported: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen und nicht-kommerziellen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen und das Werk bzw. diesen Inhalt auch in veränderter Form nur unter den Bedingungen dieser Lizenz weitergeben | |
| Identifikatoren | 10.5446/20226 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache | ||
| Produktionsort | Bilbao, Euskadi, Spain |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
| |
| Schlagwörter |
00:00
MehrwertnetzProjektive EbeneVorlesung/Konferenz
00:32
DatenflussZirkel <Instrument>MereologieGruppenoperationPhysikalische TheorieComputeranimation
02:01
IterationMultiplikationsoperatorProdukt <Mathematik>InternetworkingQuelle <Physik>MathematikZeichenketteComputeranimationVorlesung/Konferenz
03:07
Fächer <Mathematik>Metropolitan area networkGewicht <Ausgleichsrechnung>ResultanteMAPProgrammbibliothekKartesische KoordinatenDatenfeldFaserbündelDokumentenserverSoftwaretestProjektive EbeneMobiles InternetMetropolitan area networkZahlenbereichGeschlecht <Mathematik>Prozess <Informatik>KomponententestDemo <Programm>KreisbewegungSoftwareentwicklerDatenverwaltungFront-End <Software>Abgeschlossene MengeComputeranimation
06:11
GrenzschichtablösungVerzweigendes ProgrammFrequenzCodeSoftwareentwicklerComputeranimationTechnische Zeichnung
06:48
SoftwaretestPunktMereologieVerzweigendes ProgrammMathematikKomponententestProdukt <Mathematik>SoftwareentwicklerVisualisierungKartesische KoordinatenCodeRechenwerkMini-DiscKonditionszahlRechenschieberFehlermeldungInteraktives FernsehenGraphSichtenkonzeptEinfügungsdämpfungDatenbankIntegralComputeranimationFlussdiagramm
08:47
Kontextbezogenes SystemProdukt <Mathematik>Rechter WinkelProgrammierspracheComputeranimation
09:22
ARM <Computerarchitektur>TestdatenMetropolitan area networkDesintegration <Mathematik>Plug inExplosion <Stochastik>SoftwaretestPortscannerMatrizenrechnungTaskÜberlagerung <Mathematik>MittelwertProgrammierumgebungVerschlingungFront-End <Software>Stochastische AbhängigkeitElektronische PublikationSpezialrechnerCompilerLeistung <Physik>ServerInterface <Schaltung>W3C-StandardRemote AccessGenerator <Informatik>Arithmetisches MittelComputerspielKonfigurationsraumServerPlug inProjektive EbeneMultiplikationsoperatorInstantiierungPhysikalisches SystemPlenoptische FunktionDatenverwaltungSoftwaretestBenutzeroberflächeDatenfeldZweiDifferenteStellenringSchaltnetzRechter WinkelBildgebendes VerfahrenTaskKartesische KoordinatenVerzweigendes ProgrammBildschirmmaskeCASE <Informatik>SichtenkonzeptMusterspracheMereologieRechenwerkÜberlagerung <Mathematik>MAPGruppenoperationProgrammierumgebungQuellcodeBitrateRPCMigration <Informatik>DokumentenserverDateiformatGebäude <Mathematik>BitAutorisierungMatrizenrechnungTemplateComputeranimation
18:22
GrenzschichtablösungE-MailStatistikCloud ComputingIntegriertes InformationssystemMetropolitan area networkVerschlingungReelle ZahlLogik höherer StufeMigration <Informatik>ARM <Computerarchitektur>Vorzeichen <Mathematik>SicherungskopieSkriptspracheOffice-PaketGravitationsgesetzServerHauptidealringCASE <Informatik>Migration <Informatik>Elektronische PublikationZahlenbereichDifferenteInstantiierungSoftwaretestTabelleSchlussregelRechenwerkKonditionszahlVerzweigendes ProgrammVersionsverwaltungMini-DiscOffice-PaketAggregatzustandMultiplikationsoperatorRechter WinkelWort <Informatik>Geschlecht <Mathematik>Kartesische KoordinatenMathematikPhysikalisches SystemAnwendungsspezifischer ProzessorBitrateMAPKeller <Informatik>DatenbankSoftwareentwicklerBitComputeranimation
25:00
SoftwareentwicklerDatenflussEuler-WinkelArithmetisch-logische EinheitAusgleichsrechnungMixed RealitySoftwareentwicklerRückkopplungIntegralProdukt <Mathematik>MultiplikationsoperatorProjektive EbeneProzess <Informatik>TopologieKartesische KoordinatenArithmetisches MittelAlgorithmische ProgrammierspracheCASE <Informatik>SoftwaretestLeistung <Physik>AppletComputeranimation
27:25
MultiplikationsoperatorCASE <Informatik>SoftwaretestInstantiierungMAPProzess <Informatik>IterationForcingVerzweigendes ProgrammComputeranimationVorlesung/Konferenz
29:17
Demo <Programm>VektorraumSoftwaretestOffice-PaketDatenfeldSoftwareentwicklerDean-ZahlKartesische KoordinatenZweiComputeranimationFlussdiagramm
30:24
ServerAlgorithmische ProgrammierspracheProzess <Informatik>SoftwareentwicklerMAPSystemverwaltungProjektive EbeneKonfigurationsraumSicherungskopieInstantiierungSoftwaretestMultiplikationsoperatorSkriptspracheDatenbankLogischer SchlussTouchscreenKonfigurationsverwaltungKartesische KoordinatenRichtungComputerspielStapeldateiWort <Informatik>CASE <Informatik>TropfenProdukt <Mathematik>ZeichenketteVorlesung/Konferenz
33:46
Metropolitan area networkMehrwertnetzGammafunktionUniformer RaumVorzeichen <Mathematik>Kontextbezogenes SystemComputeranimationVorlesung/Konferenz
Transkript: Englisch(automatisch erzeugt)
00:00
Welcome him. Hello everyone, I'm Wojtek, I'm from Poland, from company StxNext, and today I would like to tell you about how we implemented continuous deployment in one of our projects.
00:25
If you have any questions, just interrupt and ask me. I will be happy to answer. So, just agenda. In the beginning, some theoretical introduction,
00:47
then more about workflow, so more about process, how to work. In the third part, about tools, what tools we use to implement continuous deployment,
01:00
and the fourth part, I could say the most interesting, our failures and how we manage it. With some tips for you, maybe useful for you.
01:21
And just a little, during the third part, the tools, there will be a small contest, so listen carefully and I have small prizes. Who likes prizes? Prizes, awards.
01:43
Okay, so let's start with it. Almost, almost. The prizes, Polish sweetie, from milk and a lot of sweets.
02:05
Okay, theoretical introduction. Let's start with what's the most typical way to deploy things, especially when you work with Scrum and with Sprints.
02:23
You work for a whole Sprint, a week, two or three. You develop things and when the time comes, after the two weeks for example, you deploy. And then you start to work on next sprint, next things, and then you wait.
02:49
If in the first day of the sprint you made some change, then you wait for two weeks to see it on production. This is okay, but there is another way, much more nice for the developer, the continuous deployment.
03:13
So you deploy a feature, when it's ready, somebody is testing it, QA testing it, and when it's tested, you deploy it.
03:24
So with this approach, you don't have to wait two weeks sometimes to see the users that use your feature, but you can see it just a few minutes later, when it's finished.
03:41
And you continuously do the same. So new feature, closing it, finishing it, testing, deploying, and bam, you can see the results. This is really nice for a developer, because he feels that his work is for users.
04:06
It's not somewhere in the repository for demo for two weeks, but users use it.
04:22
So today I will tell you about how we implemented it for our internal project, RMS, in the left upper corners, the name of it. It's application for managing candidates to our company.
04:45
Our company is about right now 160 people, and as you know, the rotation of developers is quite big, bigger than in other jobs.
05:05
In SCXNX it's not that bad, but still, we have thousands of candidates that we have to manage, and to do it efficiently. So we created an application for it.
05:23
It's built with Django, Postgres, on the frontend side is Angular. There are other nice libraries, like Factory Boy, it's for managing of fixtures for tests, really nice library for example, and some other things.
05:49
This is a team, it's 25 right now, I think even 26 developers work on it.
06:01
Some of those only a few days, some of those for many months, so it's quite a big thing. And workflow, how it works.
06:21
Starting with GitHub code is there, we have just one main branch, the master, and each feature is developed on new branch. This is quite normal, we call it feature branch, and when the feature is ready, a developer creates a pull request.
06:49
So in here it's nothing new. This is more visualization of it, so we have a new branch created by a developer, he creates
07:04
a feature, he adds a unit test for it, creates a pull request, it goes through code review, and there is continuous integration, which checks the tests, and the change is in the last part on the slide, so tester checks this feature branch,
07:33
so developer don't have to in here merge it, when it's okay, but tester switch to this branch and test the application on this branch.
07:50
This is manual testing, and when the tester see that it's working correctly from the business point of view, and there is no errors, etc,
08:08
so the tester merge the branch, goes to the GitHub and clicks the green button merge. If it's grey button, so it have to be rebase, then ask developer to make rebase.
08:31
When it's merged, automatically we back up the production database, we deploy, tester clicks just one button to deploy, and then we have a feature on the production.
08:53
Okay, the tools. So, the contest. Contest is about the tools, so there will be
09:02
bunch of tools, and if you know in which programming language the tool is written, just raise your hand and say loudly what's the name. Okay? No? This one is? Yeah, raise your hand. This was the requirements.
09:38
Jenkins. We have a Jenkins, yeah, it's written in Java, in here we have four builds, so we have
09:54
PR testing, PR testing is a build where tester can build his own instance on the branch that he wants.
10:04
So he goes to the GitHub, he copy the branch ID, then he goes to the Jenkins, to the configuration, fill-ins to one of the fields, the branch ID, and hits the build, build now.
10:22
And after few seconds, few minutes, he have his own instance on the branch, and he can test it. We have also staging, this is an instance which is not updated automatically, this
10:51
is more an instance when somebody from, for example, stockholder wants to see something,
11:00
then we have just one instance for some kind of demos, etc. We have testing, this is a copy of staging but it's automatically rebuilding when the branch is merged, and we have live. We have also built in the Jenkins for the live, but it's not on the same server, the Jenkins, but it's on remote server.
11:31
What's more about Jenkins? Some tips, what you can use to make it more user-friendly for Django.
11:45
You can use plug-ins for Jenkins, like violation plug-in to check pipettes, pilings, there's a nice plug-in for tasks that will be run after the build was successfully built.
12:05
Also, matrix authorization is for better managing of what user can do, for example, if we have a live instance, the build for live instance, we can assure that the live will be updated only if
12:33
for some kind of users, some part of users. GitHub plug-in has a few things there, but we use their automatic creation of tags.
12:50
We can have also packages on a Django site, the nose and nosexcover, which creates tests that
13:01
are understandable by a Jenkins, in a J-unit, S-unit, so Java, XML unit tests format. Build-out. Great. Yeah, of course Python.
13:29
In the project, we use build-out, not virtual, because we wanted to have just the same environment whenever we build it on whatever server.
13:49
And in this case, build-out is better, it's more heavy, I could say, than virtualenv, but you can build many different sandboxes, many different environments on the same server,
14:08
because the Jenkins that we have there, it has 30 or 40 different applications there. And also, in here, you can use receives, so these are some kinds of plug-ins for build-out.
14:27
Django receives this one, creates a Django instance. Receive template is for creating configuration, so you can create, for example, local configuration, localpy for Django from the build-out.
14:48
We have received for UISGI for creating, compiling the UISGI, and Mr. Developer is a very nice plug-in for downloading source code from repositories like GitHub, Mercurial, and many different.
15:15
Yeah, that's right. This is more, almost, almost.
15:26
This is about front-ends, yeah? So the JavaScript is, it's the main thing there, so Bower and Granta are JavaScript.
15:40
We use it for managing the whole front-end side, so in here you have also many useful plug-ins that you can use. Fabric, of course Python, great tool for managing remote tasks.
16:15
So, in this case, we used it from Jenkins in the post-process build action.
16:27
So, the live build in Jenkins, yeah, runs the tests, try other things like try to migrate data with South, if everything is okay, then we have the post-build plug-in,
16:47
in which we have just one command line, which runs fabric, which updates the instance on remote server of the live server of the application.
17:04
And this one is, anybody knows? Yeah, raise your hand. Yeah, that's right.
17:21
This is not so well-known application, but it's really nice. We use it not in this project that I'm talking about, but in another project that we implemented the same workflow, but in a little bit different way.
17:42
Gallery is written in Python, and it combines some things from the Jenkins, so we can build, you have a nice web interface to click things without making comments.
18:06
So, even testers which don't know how to develop can do that. And it's also use the remote tasks, so you can execute also the same task on remote servers.
18:25
Okay, this was the tools and now the challenges. The first thing that we encountered and we implemented continuous deployment was the restarts of Apache.
18:45
So, on the server that we had Jenkins, our Django application was on ModWhiskey, on the same Apache that was serving the Jenkins.
19:05
So, to restart Django from the Jenkins, we had to restart Jenkins as well. This was stupid, so we changed it, we don't use the system Apache,
19:23
but we compile a USB for each instance and we restart only this one instance that we want. This is, the pro tip is a little bit tricky, because if you run buildout, in some conditions you can lose the PID file, PID file.
19:55
So, without the PID file, you will not be able to kill the previous instance.
20:02
So, to avoid it, we used master FIFO, disk FIFO, and this is a tricky comment to restart it, if the FIFO exists and if there is an instance which listen to it.
20:22
Next one. We had the problem with migration and the data in the instances. So, when tester changed the branch for the future branch, Jenkins ran the migration, if there is any in this future branch.
20:54
Ok, this is fine, but when tester says no, this should work a different way, so he has
21:07
to revert to previous state, so he will be able to run another, to test another future branch. So, we had two choices, or support big migrations, which are quite fine with south, but not in all cases.
21:34
When it's automatically generated migration, then it's fine. If it's written by developer, then developer also have to write back migration.
21:51
The solution in here, when we are creating a new instance on a new future branch, we remove the database and we copy data from the staging.
22:07
Pro tip in here, Postgres don't have a comment to drop all tables, you have to drop one by one, this is a comment that you can run to remove all tables in a loop.
22:32
Next one, we had a problem with migrations, the south migrations on different branches with the same number.
22:48
And when the tester merges the branches, he doesn't see anything that can happen incorrectly, if there are two migrations with the same number after the merge.
23:09
So, we added just a new file, version.txt, with a current number, so if two branches have the same, changes the same file at the same time,
23:30
then the second one will, yeah, there will be a conflict and tester will see that he can't merge it, developer have to rebase it and check the migration as well.
23:47
The south migration works quite well, but only in cases when two different migrations changes something in different places, in different tables.
24:04
If two migration works on the same table, it doesn't work that well and we had problems in these cases. Last one, if developer thinks that something can go wrong, then it will definitely go wrong, so you
24:31
have to be prepared for it, make backups, create stacks, so you'll be able to revert to previous stack. And we also had just another rule for testers, upgrade only in office hours, so there will be somebody in the office who will help you.
24:57
And of course monitoring, for example, New Relic, Nagils.
25:04
Okay, this is the last one. You could wondering why Jenkins, why we want Java there, why power grant with JavaScript, why not use everything with Python, because we wanted to make it easy and low cost.
25:28
What does it mean? We already had Jenkins there for continuous integration, we already had power grant, we already have Fabric, so it was more about reconfiguration of everything and
25:44
changing the workflow, the procedures for developing code, not to rewrite all the things around the project. So we used this kind of tools. And it works quite well for our non-critical application, so this is, as I said, internal
26:12
application for managing candidates, and if we had some problems, we had to downtime for few minutes and it wasn't a problem for us.
26:26
What are the benefits that we filled very well? New features faster on production, so users, in this case the recruitment team, could use it after hours, not after weeks, and also developers had faster feedback from them.
26:51
Also, less boring stuff for developers, more things for testers, and we also felt that the feedback is good.
27:08
And just a last reminder, remember about Murphy's laws, everything can break, so be aware
27:20
of it and try to think one step forward. Thank you very much, this is it.
27:40
Time for some questions, so please raise your hand. Thank you for the talk, it was very cool, I learnt lots of things.
28:01
You were saying about PR testing, I'm not familiar with PR testing. What's PR testing? PR testing, this is a name for pull request testing, so in this case the instance works on the same branch that is in the pull request.
28:38
Thank you for the talk. In continuous deployment, the idea is to automate the whole process and just putting away all the human interventions.
28:48
And from what I see, you have deliberately put humans there, like someone is clicking on the deploy button or something like that. Is this a problem that you have faced when we are fully automating the process, we have seen lots of problems and we require the human integration?
29:06
Or maybe it was that we are not really there for the continuous deployment and we have some things to do still. In this case, we are talking about this one. This step is not automatic, the testers have to click, just click one button built live.
29:37
It's because we want live builds in the office hours.
29:45
So, why? Because application is not that used out of office hours, the main users are in the recruitment team, which is also in the office hours. This is one thing, and the second thing, if something will break, there will be admin
30:07
or other developer which can take a look on the logs or run the revert script. But yes, it could be automated in here.
30:35
How many times happens that you've got broken production?
30:41
As I can see, there is only one level of human testing. For example, in projects where I develop, we have, I think, at least three levels of testing by humans.
31:04
Yes, so we had, I don't know exactly how many deploys we had, I could say about 100 deploys, which only few, maybe five, were broken.
31:27
In only one case, we had to remove, drop the database and create it from the backup.
31:40
So, it's not that often, but still, it's better to have it some kind of procedure to get it back. Any more question?
32:08
Sorry, just following on from the gentleman's question there, did you actually automate that process of the rolling back process, or did you just do that by hand? Yes, so this is more bash script, which checkouts the, not the newest stack, but the previous stack,
32:34
runs a build-out, restarts the instance, and this is it. So, only few comments in the bash script, but you don't have to remember about it, you have just one script to run it.
32:47
But only some admin or developer have to go with SSH and do it. Thanks for the talk, I would like to know how are you managing your infrastructure, are you using configuration management?
33:08
The application, the live application, the Jenkins, other instances are on our own servers in the basement of our office, in this project.
33:31
So, in here we had full access, we could do whatever we wanted on the servers.
33:42
Any more question? Ok, so maybe I will just, this is a contact for me, if you want to talk with me, and just one thing,
34:01
this is my company, we have many projects, interesting, if you want to move to Poland and work with us, just let me know. Thank you.