Facepalm to Foolproof: Avoiding Common Production Pitfalls

Video thumbnail (Frame 0) Video thumbnail (Frame 1905) Video thumbnail (Frame 3403) Video thumbnail (Frame 5128) Video thumbnail (Frame 6924) Video thumbnail (Frame 8498) Video thumbnail (Frame 9701) Video thumbnail (Frame 10707) Video thumbnail (Frame 11599) Video thumbnail (Frame 12869) Video thumbnail (Frame 15420) Video thumbnail (Frame 16374) Video thumbnail (Frame 17124) Video thumbnail (Frame 17860) Video thumbnail (Frame 18873) Video thumbnail (Frame 19773) Video thumbnail (Frame 23174) Video thumbnail (Frame 23912) Video thumbnail (Frame 25235) Video thumbnail (Frame 25978) Video thumbnail (Frame 27419) Video thumbnail (Frame 29077) Video thumbnail (Frame 30312) Video thumbnail (Frame 32132) Video thumbnail (Frame 33810) Video thumbnail (Frame 34918) Video thumbnail (Frame 35615) Video thumbnail (Frame 36289) Video thumbnail (Frame 36878) Video thumbnail (Frame 37642) Video thumbnail (Frame 39272) Video thumbnail (Frame 39955) Video thumbnail (Frame 41010) Video thumbnail (Frame 41604) Video thumbnail (Frame 42398) Video thumbnail (Frame 43154) Video thumbnail (Frame 44118)
Video in TIB AV-Portal: Facepalm to Foolproof: Avoiding Common Production Pitfalls

Formal Metadata

Facepalm to Foolproof: Avoiding Common Production Pitfalls
Title of Series
Part Number
Number of Parts
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
"WTF asset pipeline?" "What are all these errors?" "Why is my app running so slow?" If you're new to Rails development, or just want some tips on deploying and running in production, this is the talk for you. Relying on real-world experience as part of the Heroku support team, we'll talk through common issues (and a few funny ones) we see when people take their "but it works in development!" app to a production environment.
Axiom of choice Process (computing) Computer animation Link (knot theory) Data conversion
Area Mathematics Computer animation Square number Remote procedure call Family
Slide rule Group action Service (economics) Computer animation Real number Moment (mathematics) Total S.A. Office suite
Server (computing) Multiplication sign Moment (mathematics) Horizon Power (physics) Product (business) 2 (number) Web 2.0 Optical disc drive Computer animation Right angle Quicksort Booting Absolute value Resultant Spacetime
Point (geometry) Server (computing) Game controller Context awareness Concurrency (computer science) Computer file Multiplication sign Weight Bit Database Benchmark Web 2.0 Process (computing) Computer animation Website
Mobile app Computer animation Personal digital assistant Semiconductor memory Different (Kate Ryan album) Multiplication sign Operating system MiniDisc Right angle God
Mobile app Server (computing) Link (knot theory) Multiplication sign Real number Image processing Total S.A. Limit (category theory) Leak Automatic differentiation Number Computer animation Semiconductor memory Blog Operating system Cuboid Right angle Integer Quicksort Object (grammar) Service-oriented architecture Library (computing)
Computer animation Website Mereology
Web page Overhead (computing) Computer animation Link (knot theory) Forcing (mathematics) Bit Product (business)
Medical imaging Mathematics Computer animation Inheritance (object-oriented programming) ACID Software testing
Point (geometry) Default (computer science) Context awareness Multiplication Server (computing) Shift operator Computer file Electronic mailing list ACID Sound effect Cartesian coordinate system Axiom Computer font Login Product (business) Revision control Preprocessor Medical imaging Computer animation Volumenvisualisierung Authorization Right angle
Rotation Trail Mobile app Standard deviation Server (computing) Scaling (geometry) Divisor Multiplication sign Moment (mathematics) Streaming media Function (mathematics) Login Product (business) Wave packet Uniform resource locator Process (computing) Computer animation Core dump Right angle
Revision control Scripting language Mobile app Server (computing) Computer animation Projective plane Data conversion Parameter (computer programming) Error message
Scripting language Point (geometry) Server (computing) Multiplication sign Entire function Connected space Revision control Arithmetic mean Computer animation Single-precision floating-point format Cloning Right angle Fiber bundle Table (information) Plug-in (computing) Computer architecture
Goodness of fit Computer animation Software repository Multiplication sign Gradient
Computer virus Mobile app Graph (mathematics) Key (cryptography) Software developer Real number Database Revision control Computer animation Integrated development environment Bit rate Computer-assisted translation Hydraulic jump
Human migration Subject indexing Computer animation Database System call
Subject indexing Computer animation Structural load Factory (trading post) Right angle Table (information)
Performance appraisal Game controller Digital photography Computer animation View (database)
Medical imaging Computer animation Integrated development environment Structural load Musical ensemble
Standard deviation Functional (mathematics) Software developer 1 (number) Bit Login System call Digital photography Computer animation Bit rate Personal digital assistant Video game Website Object (grammar) Associative property Library (computing)
Game controller Digital photography Word Dependent and independent variables Computer animation Multiplication sign 2 (number)
Medical imaging Message passing Game controller Dependent and independent variables Computer animation Personal digital assistant Weight Internet service provider Self-organization Object (grammar)
Dependent and independent variables Email Game controller Service (economics) Process (computing) Computer animation Real number Set (mathematics) Bit Mass
Game controller Digital photography Email Server (computing) Process (computing) Service (economics) Computer animation Code Multiplication sign Authorization Parameter (computer programming) Line (geometry)
Computer animation Computer-assisted translation
Server (computing) Computer animation State of matter Multiplication sign Line (geometry) Lattice (order) Mereology Product (business)
thank you for all and that was the end of the but so far right where it started at the IAEA have the unenviable job of speaking last today which means you're tired you want to go to sleep you haven't had an cake of were all your copies crashing so I am nervous souls do this turn to the person next to you and say I promise you I will stay awake for this entire talk a way that it and that's while our conversation as expected now turn turn your 2nd choice the gated 1 and I 2 and say that you know the guy on stages is homeless it's not proven why you say that about me but I will get to that 2nd hymenium is John party uh he wrote I help people fix things there and not homeless what kind of I and this is my house are the link is bitterly
PAT dead alive last rails comp 16 there is some history of that thank you I thought I saw was a homeless jokes no that's my past right so
and there's a reason I'm telling you this that so I actually last year a year ago year change I sold my house and I moved into a 100 foot square foot so 180 square foot trailer and I've been living on the road for about a year now and it is that sounds kind of scary a lot of people and then you throw in 3 kids and wife and RDF in that area as a whole and say as they family but it's really cool you to travel around work remote for her roku about 50 % remote and 6 cent engineering is 50 % our company and we had to
go some really cool places but that in Washington State last summer this is my
data home office is my best year all right on my floor panels my roof and and I tell you all this because I love traveling which is in a service of 4 what recede here so tempted to play the data a how
OK are animals slides coming next to the mirror but were in with where I tell you that I love to travel and in the light of today's my my talk today is basically were it covers some really common things that we see accurately support and all of these issues were and talk about and the face from moments yet was now it's fine and good and all but their face palm moments did these moments for you you you think might happen so slow what what's going on it's from a group this fall and and you have good a total of real yeah yeah know them in this is OK
so the face from moments I know this was to be that that's cool it's cool the the at the time but I can't tell you who made these some of the money that I made early on writing rails and a lot is a really common customer issues that they right and things are going right and he's canister to poke around here are some questions a lot of power yes this really simple things so but I can't tell you the odds were it would tell this entire story using a made up company by light travels to really go with a travel theme but I think you're B and B is a fantastic so use all the time but they haven't really looked beyond the horizon literally into space for space rentals so warrant take advantage of that and then wanting careless B in the and when you start at if you are a B C entactin afterwards your us another very BBC and there's a logo we're launching a now think very much a
hands of things are going OK in the last 10 seconds we want and that's just poking around the of things are going and it looks like my real that is due in about 40 requests per 2nd worst capable of around through anaplerotic ABS find people send us a results all the time and 40 isn't that great and so sort to poke around wonder what's going on in well it's let's go and of losing the server when it boots up this is what's going on here anyway knows what it I just do that in the previous 2 rails 5 what's going on what I'm doing yeah I'm doing this right there shocking shocking amount of people running Weber can production and so hopefully that's not you if it is if you if you don't know what web server running the you the running Weber the face the don't do this with sucking up to
my request but to my is the web server we recommended Broekhuis we used to do is recommend unicorn and then uh you is great for a long time it's susceptible the slow request it's actually kind of stay away from that and and there's a lot of other great things it will do you go the benchmarks on the website the fantastic and and we'll talk a little bit about to that kind of the king that that Puma gets is that it's Friday if you're using MRI which doesn't really in with points and but in reality most Rails apps or I'll bound so while you're at is out going to databases can take advantage of France uni-marts MRI so too is fantastic really easy to get started further gem file booted up really simple
and this is so this is the whole controller or not doing any kind of context switching and immediately I get double the throughput of that so if you're wondering why you should do this it's mainly because Weber has 0 concurrency technically does but in rails it has no concurrency what that means is is it can only handle 1 requested time which is really terrible and because while 1 request is coming in and getting processes that 1 comes in at half the weight and blocking with unicorn fully you're able to fire at multiple workers to handle multiple requests at a time so that is a fantastic thing were done for
great great talk around them yes thank you thank you so much lower it is it but you're to see if the and for all these in a lot of our questions and uh that we get a rope to actually start with my app is running really slow and and that is 1 of the hottest places to start of helping someone debug what's going on because almost every single thing I talk about Russell talk starts with this is really slow and you have to start poking around and figuring out what that what the reasoning is in this case uh
why poking around I solve this is accused of profits and the red shows me how much memory of used in the blue shows me something that I don't want see which is my happens in swap now swap it the 1st time I learned about swaps anemia Linux book about this that that I'm not gonna cover that only give you really quick explanation you don't know swappers I didn't when I started and so you're operating system is going to have a certain amount of RAM and and after that it's going to once that fills up as to start saving stuff to desk at your disk is way slower than in memory so when you're at starts you going to swap you received the performance there there's a couple different right side things God memory
enemies this to move main things that we usually see the 1st 1 is on is need to link some you slides after and a memory leaks are kind of nasty and bargaining take on much longer time to talk about the of these 2 links the 1st 1 is a a library pirated Stephen only as a were with the broker and it's a fantastic little thing to a run against your happen you can actually see if you do have a memory leak and so if you are concerned about that that that he also wrote a fantastic blog was over coach on debugging really and the 2nd thing we hopefully number 1 doesn't apply to the more common thing that we see actually is really simple fax people are or running too many of those workers got uniformly at tumor and they just they this How many workers should I run I don't know 5 is a whole number has low fat without them but they don't think about it was a number actually especially when it applies to memory so you look at rails evitable real that is gonna run out of the box right around 100 million 200 meds face if you've got a gig of RAM you go on a thousand 24 divided by 2 of all run 5 workers the problem is now you right up against that limit you have no headroom that your app needs to actually use a allocate more objects this you're doing any kind image processing that's the blue not in all this in your sort once walk so the the the the the better way to look at that is to just cut back anywhere processing how your memory is your memory looks after that and and then start to adjust from there but when you start going in this in the swap even have a bad day and
really he's a check this out actually is going your server context-free dash and you're going to get that number which is you were a total amount of memory so we a gets but Europe operative system has allocated so saving of 511 ads for so and right now only using 7 so just ignore the other stuff is whenever you care about and when that number that's 7 starts getting really deep and you need to be concerned about obviously there's a lot of other tools allow us to help you visualize this stuff and but 1st place so what we have is running slows check your memory usage and adjust accordingly they're moving no about
people want Cheney just fossils the from a great place on Mars 1 bedroom no problem because we've got a bad day
yeah In he discovered a unprotected part of my website and has taken the money now sits on fire this is why again more common than you
think so we put up as itself the everybody goes to all the links that we posts have dual S. at the end why people going here because the dead and you didn't tell them otherwise and so therefore they're going to this check out on the page and it's completely unprotected and really really really really easy to fix this in rails actually and we'll talk about having that rails and into next to the
first one is this is all you have to do I don't know why this is not my fault I go into production ever be go to force SSL and truth and you're done but and this will take any request that comes in on 4 80 and immediately redirect the entire thing that's HDPs now the problem with this so is that it's going all the way into your rack uh it is kind of a lot of overhead for just doing a redirect so you can use a little bit higher up in the next and was among 48 somebody comes in a son and a 4 4 3 and you can do this a lot higher up engine x is much better placed
this and will not be a few were covered at 14 that's right we are gonna change a logo rocket ship didn't didn't test well were intended to a comment super easy to do so we change it will push it and our love gone
nobody knows where when I the image should be i G can I have I over designer probably that the the birth why did this happen or a friend the acid pipeline
again against a lot it gets a bad rap and and it's unfortunate that I love the as by point if you ever had to deal with acid cashing in but before the US a pipeline with the question mark crap so much for that so this image by logo here that put is actually turn into this M D 5 the a and the fire of digested uh biomine which is no less detached costing a lot easier and if you don't know what's going on here basically what the gas pipeline will be was it looked inside your image and if it has changed it will generate new files which is how we do not have crossing the problem is is that somebody did find G and it is directly linked to the logo but instead again release effects use image that helpers and this 1 is actually as common as the CSS 1st this is way more common is a lot more so when you got your CSS stop using the URL and linking the assets because this will not use gasified when all but very few actually seen those but there is all of CSS helpers and inside of CSS other rails provides an image Majuro actually do all of the all of that 1 for you and is also if you got a fonts folder define euro acid this axiom of this the base tacitly were as well and good but make sure that you're using this for every you can use a rescue team to always uses of designers do like my old team of this is kind of what we would work the design never they're doing any kind of friends that it is to make sure that is this really tempting just to plot and the old URL other lower the precompilers but thankfully rails think for which version is uh were elected warn you if it's not in the precompilers look for 1 and rails a warning if you try to render another day far and it's not in this list but be aware of this list so if you're trying to by default render application that Jason Application of CSS if you want another 1 make sure to add to this list so it gets precompiled it's another thing we see people run into sometimes is it works locally in the push to production hasn't precompiled author Anna article about
and so we ask you to happen a lot in I not wanted twice was logs right shift allowances we do that we gotta production not long and the problem is we have multiple application servers this is a problem because now I have multiple production lot lots and I've got I've got really really big production dialogs as we've got a little problem here and if you ever if
you've never read about full factor apps I highly fajitas URL and what's the sticker look it what it says about logging it'll factor of 12 attract never concerns itself with routing restoratives output stream it should not attempt to write to or manage LOD fast instead the training process right to the mainstream unbuffered a standard out what this means is your app should not be sending stuff to production outlawed it's really easy i've seen lot apps and where and usually non roku and will despite a fall of like were only can be running 1 server forever who cares scale and so this will get to it later and then just dump everything in production of log lots at times don't and log rotations of this thing starts growing out of control and in your rails that your Rubia use standard at that
if you like to standard out now you're doing would fall factor I should do and you're just every the standard out and getting way this allows you to consume standard out in a much better way of but do 5 Splunk and use an ad on like log entries a paper trail that consume those logs and have a much better understanding of what's going on inside your moment were 7 gesture
chitchat check every last year desired appointed
script further finally sleep for so this is an actual this is an actual research script somebody gave me so I took over like a little of friends friends project in the former engineer so this is our research script what you think happens when a user comes and we're sleeping when should even be sea and engine next 503 error it's terrible and the argument was we shouldn't do restarts because you could have 2 different versions of your app running based on you can restart but that's a whole other conversation but this is that what we need is an actual legitimate way to deploy at the does involve me SSH into a server and running this so but so
you may have heard of Capistrano and I hope you for a minute if you have not detailed meaning is my favorite sorry my 2nd favorite way to deploy rails that table my 1st 1 2nd I mean is fantastic I started writing are started deploying with Capistrano of years ago and I found mean of me is really similar Capistrano if you're used to using Kapustin take a look at this and is a great plugin architecture for restarting things like unicorn who marked and inside spot great plug ins for this bullet agree on meaning is really really really facet of the same kind of idea were able and to a shallow clone Adam did make a new version folder in and do this and linking all and bundling in the migrate all that stuff for you and book what it kind of does even cooler is versus Capistrano which fires off every single communities on ssh tunnel mean will look at the entire thing bundle up the entire script open 1 SSH connection to all of your servers individual and run the whole thing on the server so the way faster and if you've never seen it take a look at this is what it looks like when you
deploy other similar to something else of history that and it will make the path from the get reproduce all the business and right the animals similarly can restart the server there's another way and this is what looks like a Baroque who uh create eurac get pushed through canasta whether it's this or whether it's using Nina and you need to have a good way to deploy if you want to ship frequently and the teams that are usually unable to and deploy quickly are teams that struggle with these 2 point practices and they think that get to at some point and they never do I get stuck because everybody's too busy doing something else and then at by the time they needed it's almost too late and they can deploy quickly so far have a way to deploy quickly so you can ship
good next problem somebody got into all over S 3 images and swap the out yeah so nefarious stay can only beautiful castles pictures and put in something less than appealing how could this have happened
the a long time ago someone decided to put our S 3 grades into the git repo then something happened maybe somebody for two-party have only recently um took the aperture and get how of the deleted it but it's still there and you get history and there was a there's a couple years ago where you you could actually I think you can still be the suddenly there's a search command be good but there's a search command will let you go through all get have been looked forestry grants and there's still they're still out there people have a secrets all over the place uh changed at 1 time right yet that guy right there that that how how having addressed rebuild morning 3 grand and so don't do this but but but but but there so much better ways of
doing this and there is something called secret start you know instead of rails now this is my is this 1 works pretty good I have a better way but if you like this version you go this route and throw your he's inside of secret stadium all these get ignore this file and and then you can use it inside a reals by calling it thusly personally I much prefer and virus and this is where all of your PC to be in and bars you don't commit them they're different per environment the running on and really easy to do it in in development you can use gentle . handrails you make a dot and file ignore it the data key values and then under OK it's really easy for a book instead advocate and then in your app this ball and and whatever the name is Cat that really easy to this is the right way of storing credentials please do it's really tempting you're going really fast the beginning of an apple like understand the gravel from here not do that later and start the correct way by using an bars way up front yeah are at
15 minutes so hurry up someone questions but they're great this is my database graph for this is my app revenue role of the orange jump rate there is my database this is no good what on earth could be causing us we're yet small overlap and I was looking
abuses by the user name this call to 2 . 3 moles its but now last 20 minutes for us being up and running we have jumped to 231 thousand users and suddenly that call is 51 ms who would like to tell me what's wrong the indexing thank you so that little thing is now this big thing and that's the only gonna get worse because we do not
have an index on using but this is 1 of my own I bet I but this would really hard early on and and this is really really easy to use a database and this index is a basic tell your database this column is something it's important to me I want the index of because example make up calls against this column frequently it's so easy do rails and you based discrete migration at the index on the table and call me want and all of a
sudden this 51 ms actually factories done a 1 millisecond the same table load but 1 asterisk here is
composite indexes it in what this new and suddenly I wanna look up by the 1 thing in and something else that original index that you may no longer applies and you need to create a composite index again really easy doing rails past an array of column names you now have a composite index right to more blow
through these fossils so close to a billion dollar evaluation the great I got my 1st
problem that I have here and start seeing the apples will slow and I've got this controller here iterating over photos and then in my view I am staying up the photo use so this is what
my what looks like so I'm getting all the images and then i've got all these user loads what's going on it is a and was 1 of then about
this music and these are really easy to to ignore wire right now and because again your your development environment is taking on the load your production environment will so all of a sudden what works locally is is getting thrashed by vitamin users you have an n plus 1 basically what happens is when you
call an object that has a relationship you start iterating over this and calling the related the associated object and so really is fix in rails all you have to do is is of going down just photos all offered photodetachment what everyone else and includes OK it can it can mess with you because it included is actually a standard in library method includes will tell Active and this is an association of what you load and what happens is is really smart thing where it goes from the 1st that's so now I will make only 2 calls the photos and then do and find all of these ratings on those photos and grab them all in 1 shot which is the way that in development is a gem called bowl at and this will show you when you got and plus ones that looks like this in your logs and it will actually tell you add this on your Finder's if you got a photo go it says right here is associated for users to add this to your Finder include users and life of the magical last 1 my
users are complaining that this things it slowly doing certain functions specifically adding photos but in of if you if you look around lecture find plenty of and of cases where you can even see were MS of will actually affect users happiness people will will be Bailyn websites because things are just a little bit too slow so this is actually pretty important to us now I
take a look and you know and I see my have pretty snappy it's of this 1 thing called photos controllers almost 5 seconds so I dig into
it and I think that's and spending almost 60 per cent 66 . 1 per cent of my time dealing with S 3 so what's going on here this is what's going on so a
request is coming in hitting the controller now it's creating this photo it's going artistry waiting for us 3 which means the user is reading for us to do something that is really that uses only care about yet it wants to upload a photo engulf what it was doing and so it's now that only rest 3 word returns a response so we're talk about
backroom workers here just a 2nd let's talk about these cases just to SMS as a missile simpler went on to do not guilty image magic and you follow them and so sigh but as a massless do we get this here the user
objects where to send them a message with provider so the question then is control organ weight
for freshness provided before return a response
this is it this is a is a great example for when you warnings about the worker actually in better 1 is a little bit supported talk about is is mailers and if you go on the reals guys yes you can see I think they use an active job actually should use mailer from a top-order semester but Miller's is a great example anything where you have to wait for external service to return a response for some it's non-critical sending an e-mail sent if it can wait a 2nd set of tobacco worker don't make your users wait for OK so so
the user that sentiment as a mass in our controller it's actually do the job I recall perform later which is the active job method and you can use psychic were 1 and the active job as it is users perform later in you pass your audience and
now instead of this that's so we go to the control or it goes and fires authors that adds the job to the human rater's which is super fast and gives the user back to what they're doing now the semester out and do whatever it wants this is what a basic job looks
like and so it inherited from acting job base you just see a performance that any arguments and then you can do whatever was costly before uploading a photo singer SMEs Jobson e-mail whatever it is this is a really really small bite-size code and saves you a ton of time and what's even better on on on the act provisioning of servers for a so you don't need as many f service now because they're not all waiting around and consuming resources they can actually get the user back and go below worker to care that which you actually need less servers so this is a great thing for the bottom line is
well the that that we sold we made a billion dollars and often the terrible idea but mostly because we fixed all the problems that we talked about the stock waxy pivoting we're going to be
building a tender at 4 cats it's going to get I don't do it but so that the
bottom line here is that if you are a new or if you've been around for a long time and when you're running a real that production and you will break the things and the important part is when you do that fail on something something does learn quickly and continue learn there's so many things that are changing in Wales there's a meeting the changing the state of the art of the servers you're running and don't be discouraged keep learning and you're gonna be great but thank you very very very very much higher as a result of a real world and some of the them all