Packaging and Shipping Rails Applications in Docker
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 44 | |
| Number of Parts | 89 | |
| Author | ||
| License | CC Attribution - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/31540 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
RailsConf 201644 / 89
1
2
3
5
7
8
11
12
19
22
28
29
30
31
33
45
47
50
52
53
58
62
67
71
74
76
77
78
79
80
81
84
85
86
88
89
00:00
Data storage deviceGenderProduct (business)Installation artAreaCASE <Informatik>Cartesian coordinate systemOrder (biology)Integrated development environmentEquivalence relationFormal languageSoftware developerCodePhase transitionBuildingBoiling pointSoftware testingInheritance (object-oriented programming)Multiplication signDoppler-EffektPoint (geometry)Process (computing)Medical imagingMathematicsMathematical optimizationBitContent (media)AuthorizationFamilySocial classInstance (computer science)Channel capacityComputer architectureDecision theoryAxiom of choiceScaling (geometry)Analytic setPhysical systemGoogolLimit (category theory)Operator (mathematics)Buffer solutionPresentation of a groupVirtual machineWave packetVacuumQuery languageThread (computing)Line (geometry)Metric systemStreaming mediaPower (physics)ExpressionGame controllerReal numberFeedbackLogicRepository (publishing)Server (computing)Binary imageBinary codeDatabaseView (database)SoftwareFiber bundleTunisMemory managementEndliche ModelltheorieResponse time (technology)MereologyComputer fileObject (grammar)IdentifiabilityKeyboard shortcutWindows RegistryRoutingInterface (computing)Mobile appCode refactoringFocus (optics)Computer animation
09:38
Process (computing)Repository (publishing)Musical ensembleMedical imagingDoppler-EffektContrast (vision)Physical lawMultiplication signParallel computingMathematicsFiber bundleState of matterCompilerEndliche ModelltheorieCausalityServer (computing)Type theoryOperator (mathematics)Bit rateInternetworkingSoftware testingINTEGRALSeries (mathematics)FeedbackOrder (biology)Computer fileCodeCartesian coordinate systemACIDHelixIndependence (probability theory)Directory serviceBuildingProduct (business)Execution unitDivisorIntegrated development environmentDatabaseUnit testingBuffer solutionFrequencyContent (media)NumberView (database)Keyboard shortcutBinary codeDemosceneMobile appCache (computing)Flow separationParallel portTask (computing)Installation artTerm (mathematics)2 (number)CompilerRoutingSinc functionReal numberData recoveryComputer animation
19:07
WebsiteData managementPoint (geometry)ProteinRandomizationComa BerenicesType theoryCartesian coordinate systemVirtual machineOperator (mathematics)Open sourceProduct (business)Dependent and independent variablesNumbering schemeExecution unitDecision theoryBitMathematicsDialectLevel (video gaming)InformationComputer fileServer (computing)Service (economics)Integrated development environmentMedical imagingAbstractionProxy serverBasis <Mathematik>DatabaseDegree (graph theory)Video GenieFrictionSet (mathematics)Software developerFigurate numberMereologyFiber bundleNumberFreewareProcess (computing)File formatFactory (trading post)Arithmetic meanMultiplication signMixed realityStreaming mediaMobile WebComputer architectureCuboidDoppler-EffektRepository (publishing)TwitterCache (computing)Windows RegistryDifferent (Kate Ryan album)Random matrixConfiguration spaceMetropolitan area network2 (number)Fault-tolerant systemWater vaporMobile appLecture/Conference
28:35
Computer animation
Transcript: English(auto-generated)
00:20
I'm Alan Espinosa, and I'm a support engineer at Engine Yard. Today, I'll be talking about the aspects of deploying Rails applications with Docker. My background is that I'm an
00:43
engineer. I'm mostly from my day-to-day work, so I've used Ruby back in the day when it was still popular to build systems before Go came along, but it's still my go-to language of choice. I'm not really a Rails developer. My last time I used Rails was six or eight
01:05
years ago, I think. I do a lot of Ruby development, not just with the web. I'm the author of the Docker high-performance book by Packet Publishing. Given Docker changes a lot, the book will
01:25
probably have obsolete content the next few months. When writing the book, I tried to make sure what are the concepts that can last through the Docker updates. I'll do a bit of talk
01:43
of that in this session. I'll be talking a bit from the second chapter of the book on how to optimize Docker images. Then, I'll tailor it so that you can figure out how to do your
02:06
deployments using Docker with Rails in mind. Optimizing the way you roll out Rails in Docker. When we say about optimization, most of us think about having faster response
02:27
times and having your controllers respond really fast. You have asynchronous workers spinning up those things. All in all, if you look at the broader picture, performance is all
02:41
about improving the experience with our customers. From the experience of our customers, you trace the value stream. Then, you can start doing things like refactor your controllers, your business logic based on the feedback you received from basically production traffic
03:03
when interacting with your customers. Another way to optimize down the line is you can tune the middleware. You can figure your unicorn workers or Puma threads, and then set the memory allocation so that you're utilizing your machine. You can tune the SQL query
03:25
so queries are fast. All this tuning, you don't operate in a vacuum. The tuning you make is informed through your instrumentation in your application and your machines. You put in logging
03:47
and then application metrics where they say my application is okay, how users are interacting, so even something as simple as Google Analytics can give you a lot of insight.
04:02
Then you can correlate that with system metrics in which you can use that to inform your scaling decisions. You scale up your application to keep up with the demand from your customers. You add just architecture, you add caching, you add capacity. How many of you
04:24
know when you need to spin up a new instance of your Rails application? How do you know the limits? Having a good story about how you operationalize your application is important
04:43
when optimizing. You try to tune your application so that it performs faster, but in the end, you need to roll changes to your environment. If your deployment and process is slow, then
05:01
the tuning you might do now might be obsolete by the time you get to production. There's a lot of things that you can do. Here in this talk, I'll focus more about that. Even though I focus mostly on Docker, these concepts are fairly broad and general. Most
05:30
early adopters wrote in the container hype, so now we're starting to get past that point now. In the end, even though you can package Docker containers and stuff, it all boils down
05:48
to focusing to what's the value of our application. Docker is only a tool to reinforce the way you deliver. I was in our booth at Engineered earlier, and I started talking to people on
06:05
how they use Docker. Most of the time, there's a lot of, okay, so now we're just starting testing. We're trying to convince people to run it in production, and there's
06:20
a lot of resistance with that. As an operations person, I can understand some of that hesitance, so trying to know what it means to change your stack to a container base will help people convince if you really think Docker is for you because it all boils down to delivering
06:48
your app, basically. In delivering the app, normally, we just talk about deployment, but there's also the build phase. There's a natural tendency to think of the build phase
07:06
as compiling code, a binary, like you have your C or Go code to be binaries. This doesn't seem intuitive at first for Rails and Ruby developers because Ruby's an interpreted language, but if you look at how things are, there's an equivalent of binaries in
07:28
Rails. When I say binaries, it means that anything that's needed to be dropped in the environment, like in production, in order to run the application. It's important
07:44
to know what will get deployed so that in case I get paged at three in the morning, I know where to look at. In Rails, you have your gem packages, so they're a nice way to do a gem install in production, and you're done. Aside from that, it's not really
08:06
the final binary itself because when you do a gem install, sometimes if you depend on native bindings like the FFI library, you compile stuff and produce the shared object files. The code in your Rails app is part of the binaries, so the controllers,
08:26
the models, and the routes. If you don't get LS files, everything there is part of your binary. Then you have the dependencies, the gem dependencies for that application.
08:41
You do a gem install-g or a bundle install, and then finally you have your Rails assets. There's a lot of binaries when it comes to making a ready-to-run Rails app. The thing with Docker is that it gives a nice interface to wrap our brains around
09:07
because Docker has the notion of a Docker image, the container which needs to run. All those Rails binaries need to be merged into one binary called the Docker image that
09:23
needs to be deployed. You build it in your build server like Jenkins, and then you push it to what they call a Docker registry. Basically, it's an artifact repository like RubyGems where, from git commit, you build a Docker image, you push it to the Docker
09:45
registry, and tell your ops teammates that, okay, I have my image ready, you can now pull it and deploy it in. There's just one thing that changes in
10:00
your application. Basically, you add a Docker file which defines how the Docker image is built. For those of you who are just starting with Docker, this is just a basic Docker file to define the image. Here you have the environment you want to, so from Ruby
10:23
2.2, and then you add your current directory in your build. Basically, it's all the files needed by Rails, and then you do a bundle install to put in the dependencies, or you compile assets. In the end, you also define how to run your application. Here, you run
10:44
real server, or in production, you should be running unicorn passenger or Puma instead of WebRic. In the build process, we have something like this. If you run the Docker build command and specify the name, so here I'm naming it Rails app, so you can see it's
11:07
starting to compile the Docker image. Here, you see it's adding the Rails directory to
11:21
your application, and then after that, it pulls in dependency, like do a bundle install. Here, you can see it's compiling binaries, native bindings to live XML, because I have no recovery installed. Here's just a short view of what the build
11:44
will look like. When you do a Docker build, it will run for a few minutes, because you're pulling in gems, you're downloading gems, and you're compiling the gems. Here, it took one and a half minutes. A feature in Docker that helps in the build process is
12:06
its concept of a build cache. If you run the same build again without any changes to the code, the build will finish right away. Here, it just took one second. Behind the scenes, you can see that since there were no changes to your application, since
12:24
you built it earlier, it will reuse the cache to rebuild the image, same with bundle install. However, if you make a change to your application, and you make a small change,
13:01
for example, you updated the routes or you changed the model, the build will take just as long, because since in the Docker build steps, you had a new content in your application, so it created a new, what they call it, image layer. The next preceding
13:25
steps would need to be rebuilt, because the dependent one is a new layer, so it is not much of a problem if you're starting out, but once you have a lot of teams or
13:44
you're trying to do a larger factor, like trying to do a bundle install every time you make a change starts to get painful. Those one minutes will start piling up.
14:02
What you can do is you can optimize your Docker builds, wherein you separate your application on according to which one doesn't get updated and which one gets more updated. Here, I split my gem file and my actual application, so that you'll be able to exploit the cache
14:28
more often. It's the same concept as having a separate rake task for your unit test, where you finish right away versus your integration test, where you need to spin up a database
14:42
or a cache or everything else in your stack. The initial build is the same, like around one and a half minutes. It will take just as long, but if you make a change to just your application, it will finish as if nothing was changed. Well, actually, something changed.
15:05
Here, the change happened at the later step when you added your application code. If you didn't add anything in your gem file or gemfile.lop, it will reuse the cache you had earlier, so it will greatly improve the build time.
15:26
The concept of having a build process when making your Rails app is being able to get feedback as fast as you can, if the artifact or the Rails binary that's ready to deploy
15:46
is actually good to deploy. After producing the Docker image, you vet it through a series of tests, like your unit test, your integration test in your delivery pipeline, and guarantee that it's good to go. When it's good to go, you're now off to deployment.
16:06
I found this on the internet where they substituted the compiling with deploying, since deployment takes most of the time, I think, especially on a Friday night. We've had a lot of customers,
16:28
especially on a Friday afternoon doing the deploy and supporting them. We've had customers who talk to us for support where their deployment takes 30 minutes to finish. Without a rapid
16:47
deployment process, then you don't have the valuable feedback of knowing if what you change is actually useful for your customers. I'll show a few items for you.
17:18
As an operations engineer, I don't really like doing this type of deployment process
17:26
where you log in to the server and do a get pull of the latest deployment and do a bundle install. I guess it's a personal preference, but one, it's slow because there's
17:44
a lot of change. You have to pull in a lot of gems and recompile everything in the build process. Sure, you can parallelize it across your fleet, so you have Capstrand or some other thing that SSH in parallel to do the bundle install.
18:04
In terms of being able to roll out the changes safely and the ability to roll back, you want to do it little by little. Your parallelization is limited by how much you want to update
18:21
at a time. If you do a canary deploy, you want maybe to deploy one first and then the next two, three, four, until you finish your full fleet of servers. That will slow down the process. Contrast with that to deploying Docker images. Docker has a command
18:52
called Docker pull, which basically downloads the image from the Docker repository. The deployment workflow is just download the image and run it. It's simplifying your deployment
19:06
process. If you have a canary deploy, you can do that as well. You're now bound to how fast you can download your images from your Docker registry, like Docker Hub, and not from other sources like Ruby gems. I'll talk more about that in a second.
19:29
In the end, even though we rely on a lot of community packages to make our application,
19:41
it's still us who's ultimately responsible for the availability of our application. This site got popular in Twitter during the NPM left pad thing, but it's concept was
20:01
very powerful for everything. The site is whoownsmyavailability.com. If you refresh that, it will send out random articles about availability and the concept of reliability and the notion introduced by human operations. It's a nice site to check out if you want
20:24
to do reading on operation related stuff. This is a typical architecture of, well, it's not an architecture, but how the value stream goes. We have our customers relying
20:44
on our application to be up all the time because it serves their own business usage well. Then, conversely, we are dependent ourselves on other services for their availability
21:00
for us to be able to make our application. We're dependent on Ruby gems. If you're using Debian, you're dependent on the mirror of app repository. We introduced Docker in
21:20
our stack, so now we're dependent on Docker Hub to pull our images. I guess this is where your operations teammates are having their hesitance because you're adding another dependency that can cause things to break. If you're relying on these services, it's
21:46
good to be able to vendor them and not rely your deployment process on them so that even though Ruby gems goes down, Docker Hub goes down, your application can still be deployed or you can still roll out changes if you need to update things. You don't need
22:06
to declare a snow day or something like that for your team. What I like to do, even
22:20
on my dev machine, is to add proxies everywhere. There's a notion in corporate environments where developers hate configuring the proxy settings for their development environment. There's a lot of friction there. There was a talk yesterday from Jamie Riesel about
22:46
trying to understand ops teams, where they come from and what's the source of the grumpiness. Trying to understand and show empathy goes a long way, and you can also learn a lot
23:01
of stuff from our teammates in other departments. Here, actually, in my dev machine, I have three proxy servers for each type. I spin up servers, I have a mirror for app, and
23:25
then I have a mirror for a Docker registry, and I have a mirror for my Ruby gems. You can do these things in your dev machine or your Jenkins server. Bundler has the mirror
23:44
setting, which basically says if in your gem file, if I have a source, rubygems.org, it will download to another endpoint. It's like man in the middle attacking your dev environment. It produces this entry in your user level Bundler config. I also do this
24:14
one in my gemrc, so I actually remove rubygems.org in my gem sources and add my local one.
24:24
Then I install any of these proxy repositories. You have artifactory, Nexus, Docker registry for Docker images. Out of the box, Nexus and artifactory supports different formats
24:44
so I actually just install one, just Nexus because it's free. I have my own proxy Ruby gems and my own Docker registry. When I do a bundle install, I'm not dependent
25:06
on downloading images all the time. Actually, when I do development, I can do a git clean dash fd and then it will remove all the cache gems, all the compiled gems and then I do
25:21
a bundle install again and I can install it right away. Not depending on the dependencies also works for databases. If your database is down, then the Rails application should
25:42
be able to degrade successfully. Your master database may go down so people can post updates to their accounts but if you have a slave database, then the Rails application can
26:03
read from there so you can still serve the request on a read-only basis. Everything doesn't just fall down. It's bit by bit falling down but if you have a hole in the ship, you can start bucketing out the water while another part of your team does the
26:26
plugging in of the hole. It's trying to handle failure gracefully. In conclusion, even though containers and Rails abstract a lot of information from
26:43
us so that we can focus on actually writing our app, we still need a good reliable infrastructure that we can build upon. Otherwise, it all crumbles down if you have bad foundations. It may be the magic and the elegance of Ruby and Rails that attracted us to our careers
27:08
but growing as engineers, knowing the magic behind what we're using and knowing the higher level first principles can help us accommodate changes in our stack, in our application
27:30
when doing a deploy, debugging things when things are on fire in production, knowing when things fail. Knowing all these things will make us have a more operable application
27:51
environment so that we can focus on serving the needs of our users.
28:03
With that, I'm done with my talk. If you have questions, I can take them. If you go via our booth, the engineered booth, we have limited copies of my Docker book. If you pass by and you can talk to me and tell me about your story in using Docker
28:24
or convincing management to use Docker and so on. Thank you.