Supercharge your development environment using Docker
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 93 | |
| Number of Parts | 119 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/19960 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Place | Berlin |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
EuroPython 201492 / 119
1
2
9
10
11
13
15
17
22
23
24
27
28
41
44
46
49
56
78
79
80
81
84
97
98
99
101
102
104
105
107
109
110
111
112
113
116
118
119
00:00
CodeMathematical singularityWeb-DesignerSoftware developerComputer animationLecture/Conference
00:21
BitWeb-DesignerSoftware developerSystem administratorComputer animation
00:44
VirtualizationCuboidComputer animationLecture/Conference
01:01
Overhead (computing)Link (knot theory)Binary fileInclusion mapComputer-generated imageryMetropolitan area networkDuality (mathematics)LoginData managementFront and back endsServer (computing)Electronic mailing listCurve fittingVarianceHecke operatorMaxima and minimaEmailWorld Wide Web ConsortiumData Encryption StandardRevision controlComputer fileMedical imagingDatabaseIntegrated development environmentService (economics)Library (computing)MathematicsWorkstation <Musikinstrument>BitInstance (computer science)Metropolitan area networkMultiplication signProcess (computing)Cartesian coordinate systemElectronic mailing listDirectory serviceVolume (thermodynamics)FlagLetterpress printingType theoryArithmetic meanServer (computing)InformationGodTable (information)Inheritance (object-oriented programming)Default (computer science)Product (business)Term (mathematics)Link (knot theory)Disk read-and-write headKernel (computing)Profil (magazine)Repository (publishing)Cellular automatonRevision controlFile formatQuicksortSet (mathematics)Run time (program lifecycle phase)Data recoveryCentralizer and normalizerWordOnline helpShared memoryRow (database)AreaSoftwareAngleVirtual machineFunction (mathematics)Overhead (computing)CuboidVirtualizationLoginSoftware developerMobile appBootingIP addressWeb applicationConfiguration spaceCache (computing)MiniDiscPoint (geometry)BuildingDifferenz <Mathematik>Maxima and minimaCore dumpSoftware testingComputer animation
10:29
Process capability indexClient (computing)Electronic mailing listPort scannerSineBootingLink (knot theory)Integrated development environmentIP addressVariable (mathematics)Beta functionTraffic reportingIntegrated development environmentSet (mathematics)Connectivity (graph theory)Medical imagingProcess (computing)MereologyProduct (business)MathematicsService (economics)Centralizer and normalizerDemonWrapper (data mining)Level (video gaming)Projective planeCuboidAreaDatabaseShift operatorComputer configurationLocal ringWorkstation <Musikinstrument>Software developerPower (physics)WebsiteInstance (computer science)Formal languageScripting languageBlogRepository (publishing)Physical systemCivil engineeringStack (abstract data type)Multiplication signStapeldateiVulnerability (computing)Revision controlWeb applicationLecture/Conference
17:14
ArmMetropolitan area networkMedical imagingRepository (publishing)Service (economics)Product (business)Computer fileRevision controlServer (computing)1 (number)Multiplication signWindows RegistryConnectivity (graph theory)MereologyProjective planeRight angleOpen sourceContinuous integrationLibrary (computing)DatabaseBackdoor (computing)Vector spaceSummierbarkeitGroup actionPerfect groupRun time (program lifecycle phase)Local ringReplication (computing)CodeGoodness of fitCentralizer and normalizerCartesian coordinate systemVirtual machineBranch (computer science)Volume (thermodynamics)CodeRegulator geneDirectory serviceSoftware developerLogical constantSoftwareStructural loadPower (physics)CASE <Informatik>Message passingSoftware repositoryAnalytic continuationProxy serverSource codeLecture/Conference
22:31
Googol
Transcript: English(auto-generated)
00:15
Thank you all for coming. So my name is Denny.
00:21
I've been doing Python development for a couple of years now, mostly web development and some tooling and CLIs. I also do a lot of sysadmin DevOps stuff and I like to automate as much as I can.
00:41
I work for a small Python shop based in... You probably already use a full-blown VM like VirtualBox and possibly Vagrant and stuff like that. The most important thing to me is that containers are fast, so starting a container is measured
01:08
in milliseconds most of the time. You can start hundreds and hundreds of containers and that's all because there's minimal overhead because containers share the same kernel and therefore it's much easier
01:25
to access resources like disk network, etc. Also images aren't copied for every container that we run from them, rather like I said containers only save diffs to disk.
01:43
This is due to a technology called UnionFS. So all of this should make it easy to run your whole production stack locally and you should do that because it's very important for everybody on the team, for every workstation to have the same versions of libraries and databases and everything on every workstation
02:08
and this goes all the way down to C libraries. So how do we run a container? We use the docker run command. We say with the T flag which base or parent image to use and we supply a command to run.
02:23
The I flag denotes that we want to run it interactively so we don't want the process to go into the background whereas if we supply the D flag we are going to run the container in the background and notice that we didn't have to supply a command here because the command was baked into the image and it's only overridden if user supplies another
02:45
command. You can also use tags to have different versions of your images for example to denote versions of your database and so on. So you would use docker ps command to list all the running containers that you have on
03:06
your machine. That shows you among other things the container ID which you can then supply to the logs command to get the logs that your service is writing to standard output. You can also use stop, kill and start commands to restart the container, stop it
03:24
or kill it. So by default the stop command sends a term signal to the process inside the container but it has a default timeout after which it sends the kill signal. So that's a little bit about containers, now we are going to talk about images.
03:41
Basically you can build images manually and when I talk about images I don't mean base images like the Debian image that we saw before, I mean your images that are going to be based on a parent image like image for posgres or a cache or something like that. So you would run a bash prompt interactively from a Debian image for instance.
04:03
You would type in your commands to install your services, configure them and whatnot. Then you would grab the ID with the docker ps command and then you would commit that container to a new image. So the commit command creates a new image, you can then use the docker tag command
04:20
to tag that image with a version perhaps or something else. You can also rename it, give it another name. So the username prefix is important for pushing to the central hub. So as I said, a hub is a hosted central repository of private and public images.
04:41
The images that aren't prefixed with the username are images maintained by the core team. Images done by the community are prefixed with the username and you need to, before you can push an image to the central repository to be able to share it, you need to log into
05:00
the hub. So that was a way to manually build your images. A better way is to use docker files. So it's a small DSL that you would use to describe what files to add to the image, what commands to run, and what ports to expose.
05:25
Also, you supply the baked in command to run if no other command is supplied via the command line. And you just use the docker build command to point it to the directory where the docker file is located and you name your image and it will build the image for you.
05:43
So most of you are probably thinking about what do I do on a Mac because docker requires a Linux kernel. So a docker consists of a docker daemon, which is basically just a HTTP API, and the
06:02
docker CLI, the command line interface that I just showed, just talks to that HTTP API. Having that in mind, there's a helper application called boot to docker, which is the official way to run docker on Mac, which basically uses virtual box and sets everything up on your Mac.
06:23
So you can use the CLI to communicate with the docker process running in the virtual box instance. Okay, so those were the basics of how you would use docker to manipulate images and containers, but how does that help you in your development environment?
06:41
It's really important to run services in the same version of services that you run in production locally, so that way you always know if something works on your workstation that it should work on production as well. So you eliminate a lot of bugs, not just between workstation and production, but between two workstations.
07:04
So what we have here is an instance of running, an example of running a Postgres database. So we exposed a default port that the container exposes and we map it to the same port on the host so we can connect to it from the host.
07:20
If we didn't supply a host port, docker would generate one for us and configure all the port forwarding and whatnot. Also, containers themselves are ephemeral, which means that when you start a container, you write some files, then you stop it and run another container from that image.
07:41
The changes that you did in the last container are gone. This is because the image itself is immutable and you just started the new container from point zero. This is troublesome if we want to run a database, because we want to save our tables and our database data.
08:01
And the way to deal with this is to use volumes and we tell it to mount the host directory inside the container in the place where the database is going to write the data. So every time we run another container, we will still have our database data retained.
08:21
There's another slightly more portable way, which is to have one container for your data, which doesn't even actually have to run for this to work. So one container for your data that exposes a volume and then we use the volumes from flag to mount that data folder inside a running container that actually runs the Postgres process.
08:45
I like the host version more for my development environment, at least, because I like to run the clean command a lot, so I delete most of the stopped containers and this way I don't have to worry about deleting my test database data. So I've shown examples of how to run services in a container,
09:03
but what's the benefit of running your web app in a container? So basically you simplify the runtime a lot. You make sure that everyone on the team uses the same version of all the dependencies. This is something that virtual end should be for in the Python world, at least,
09:22
but this isn't true for C libraries. As everybody knows, that's ever tried to compile the install the Python imaging library, so you mostly saw warnings at the end that told you that some image formats aren't supported. And basically this eliminates that problem, so you don't have stuff working on one workstation
09:43
and not working on another because of missing C libraries. Also, an added benefit is that you can use links. Links is a feature that Docker provides for letting containers talk to each other in a way. So you would name and start your Postgres container and you would use the link flag
10:04
to link that Postgres container with this name inside your web app container, which basically allows the Django app container to be able to communicate with the Postgres container via the host name provided here.
10:27
Also, it exports a lot of environment variables, ports and IP addresses and whatnot, so you can use that to bake those environment variables in your Django settings file or whatnot.
10:44
So how do you automate all of this? Because presumably not everyone on the team needs to know how to run Docker. I use makefiles a lot, so I built a lot of targets or make commands for running each service.
11:04
So just a one-liner command to run a Postgres database or a queue or whatnot. And I also have commands that can bring up the whole development environment for that particular project.
11:20
But sometimes batch scripts and makefiles aren't enough. So remember I said that the Docker daemon is just an HTTP API. So there's a project called DockerPy, it's the official Python wrapper for that API.
11:40
It's available on PyPI and it lets you do I think all of the things that you can do on the command line interface and there are other wrappers in other languages, but to my knowledge this is the most complete one. I could be wrong, but I think it's so. So it lets you, you could write in pure Python your whole script for bringing up your whole development environment.
12:10
So but what if you use Ansible or Chef or Puppet or any other of the provisioning tools? You can basically use both of them together.
12:22
So an example, an Ansible example would be that you could use a Dockerfile to just bootstrap the environment that Ansible needs to run and then run your playbook to provision a database inside that image. Another option is to have Ansible or Puppet or Chef connect to the local IP address of the container.
12:53
But this is bad because for that to work you need to have SSH-D or another agent running inside the container
13:05
for the provisioning tool to be able to communicate with it. And for that to work you also need some kind of supervisor which can be upstart or a systemd or supervisor d which would allow you to run multiple processes inside the same container.
13:29
This isn't considered best practice, so Docker advises you to run one process per container so it's a lot easier to update just one component of your system or just
13:46
not update it but possibly change it for another component. So for instance if you had SSH-D in there, if there's an SSL vulnerability you basically need to update that whole container
14:00
which you possibly could avoid if you didn't have that process running. Also there's another project called Fig. The best way to describe Fig is
14:21
as vagrant for Docker. So what vagrant does for VirtualBox, in my mind Fig does for Docker. You can describe, you use YAML syntax and you can describe how to build your images, what ports to expose to which container to link your web app container.
14:48
It also supports automatically downloading images from the central hub. And with one command you can basically have a swarm of containers running without the rest
15:04
of the team needing to know the internals of how this works. Also I read like an hour ago that Fig is becoming part of Docker so there's probably going to be some interesting development there. It was announced on their blog. So in summary I hope I've convinced you that you should
15:30
run your production stack in development environment because Docker makes it easy. Not all of the team needs to be able to install all these services on their workstations.
15:45
You use the same environment everywhere, not just workstation versus production but on all of the workstations and it should make your upgrade process more easy because you can
16:07
easily rip out one part and change it with an updated version or a different process. And this all happens transparently to your team. They don't need to know that a new
16:22
image has been pulled from the central repository, it just works. Also I didn't talk about deployment in this talk because I don't have enough time but using Docker in this way really simplifies deployment because it enables you to bring in
16:45
new team members on a project more easily. So imagine that you or your DevOps person have set up the production using all of these images and all you need to do to be able to bring a new person up to speed while setting up their development environment is supply them with the
17:04
images that you're already running in production and they don't need to do any other setup. That's it, thank you. Questions? We do have a few minutes for questions
17:29
and there's the microphone at the back if you want to walk up to it. Hi, thank you. As you're doing Django development, I was wondering, do you include the
17:42
Django application code inside your Docker images or do you keep it separated? So it depends on the project. Some projects aren't run inside a container because they're simple by themselves. We just use the supporting services in containers like a
18:01
database. But other projects have a Dockerfile committed in the repo and on deploy you just rebuild that Dockerfile to have the latest version of your source repository inside an image. Okay, so you include the code inside the Docker image? Yeah, you use the add
18:23
command in the Dockerfile to include the directory with the code inside the Docker image. Great, thank you. I've got one question which is probably very stupid. How do you edit code
18:41
inside a repository inside a Docker container? So you would use volumes for that. So you can mount your working directory inside the running container in a known path. So when you edit the code on your local machine, it's automatically visible inside
19:05
the runtime inside the Docker container. That makes perfect sense. Another question, usually you can't use the central repository for anything other than public stuff. What do you do
19:23
with your own builds or with your own applications? Where do you your own data? Can you somehow put that into a custom or your own private central repository? So the hub does allow for private repositories, but it's not a free feature.
19:43
But there's an open source component that's called the registry which you can install, which is Python powered. You can install on your server and you can use that for pushing your private images. Also, sometimes if it's a small image, maybe you don't need
20:01
to rebuild, maybe you don't need to have it deployed on a central hub, but you can just rebuild it on deploy. Depends on your use case. Yes, the problem is, for example, with European regulations, you wouldn't be allowed to trust the Docker central repository with
20:22
quite a few of what you would want to put there. Thank you. Hi, quick question. Do you have any tips about the automated deployment from continuous integration? Like you have always your master branch which is deployable and then you deploy automatically from to
20:45
production from that master. Do you have any tips like build images or not build images, just do it on the fly on the servers? What's the best practice for that? Because I'm struggling with that right now, how to do it like perfectly. I'm not sure there is a best
21:04
practice yet. So how do you do it? Because there's a lot of solutions that don't necessarily work the same way in the sense that you can, if you get tired of one, that you can use another. That's something that the Docker team is trying to fix, I think, with an
21:26
introduction to the new library. But the simplest way I think that you can accomplish that is basically just have a deploy trigger on GitHub or whatever if the build passes to trigger the server to just rebuild the new image with a new tag, so the latest version.
21:49
And then you switch your load balancing software or proxy software like Nginx or whatever to point to that new image. That way if it doesn't work you can just switch back to the
22:01
old one. Okay, so spawning new one, new Docker container, if it's fine, then okay, persist def and the other ones like going down. Yeah. Cool, thanks. Thank you. I think that's unfortunately all the question time we have, but you can find Denny outside if you want to
22:21
ask him. If we can just thank Denny again.