I don't like Mondays-what I learned about data engineering after 2 years on call
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 160 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/33697 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 2017125 / 160
10
14
17
19
21
32
37
39
40
41
43
46
54
57
70
73
85
89
92
95
98
99
102
103
108
113
114
115
119
121
122
130
135
136
141
142
143
146
149
153
157
158
00:00
DisintegrationGoogle AnalyticsSystem callArithmetic meanMoving averageIncidence algebraForm (programming)BitView (database)System callFreewareSystem programmingMereologyMenu (computing)Goodness of fitMultiplication signComputer-assisted translationRight angleCASE <Informatik>Total S.A.NumberSoftware developerInstance (computer science)Video gameComponent-based software engineeringState of matterLecture/Conference
02:56
Ring (mathematics)WordCASE <Informatik>Mereology1 (number)Normal (geometry)System programmingMultiplication signSystem call
03:56
InformationComputer wormDean numberError messageAddress spaceEinheitswurzelLimit (category theory)Incidence algebraLibrary (computing)Group actionOffice suiteProduct (business)WindowSystem programmingSystem callEinheitswurzelMultiplication signEmailCausalityReal numberInformationWeb pageError messageWebsiteSoftware developerPoint (geometry)Data loggerProcess (computing)Selectivity (electronic)NeuroinformatikBitLevel (video gaming)Computer animation
07:50
Ring (mathematics)CausalityMathematical analysisEinheitswurzelLine (geometry)Group actionGame theoryCuboidSource codeMultiplication signEinheitswurzelInformationCASE <Informatik>Group actionSlide ruleIncidence algebraPoint (geometry)Series (mathematics)CausalityError messageData integrityExtension (kinesiology)
09:50
Game theoryMessage passingCausalityPoint (geometry)FrictionContext awarenessError messageTelecommunicationEinheitswurzelMessage passingMultiplication signGroup actionField (computer science)System programmingTraffic reportingDirection (geometry)Line (geometry)NumberBitLatent heat9K33 OsaIncidence algebraWebsiteFigurate numberDefault (computer science)
12:20
View (database)FamilySoftwareSet (mathematics)Expert systemWebsiteMultiplication signWater vaporDatabaseIntegrated development environmentMassTelecommunicationReplication (computing)Product (business)PlanningProcess (computing)Sinc functionLecture/Conference
14:20
Web serviceDesign by contractWeb serviceGame controllerMultiplication signBitMathematicsQuicksortProduct (business)Incidence algebraSpacetimeFigurate numberWebsiteCodierung <Programmierung>Point (geometry)Event horizonSelf-organizationWordPhase transitionError messageDecision theoryInformationGoodness of fitWeightLecture/Conference
17:25
Medical imagingCAN busRing (mathematics)Group actionWeb serviceProcess (computing)Game controllerEmailComputer programmingMultiplication signMessage passingBucklingView (database)System programmingBitDemosceneNetwork topologyComputer animation
19:28
System programmingGraph (mathematics)ChainHoaxIncidence algebraNormal (geometry)Wave packetMultiplication signSystem programmingOrder (biology)LoginBefehlsprozessorWeb pageInformationView (database)Hard disk driveProcess (computing)MiniDiscPhase transition
21:32
Source codeNormal (geometry)Bit rateSystem programmingMeeting/Interview
21:54
Associative propertySoftwareData qualityQuicksortNumberReading (process)System programming
22:44
Covering spaceSoftware developerData conversionTwitterDecision theoryBlogElectronic program guideProduct (business)Software bugSoftwareMultiplication signInformationSystem programmingString (computer science)Dependent and independent variablesConnected spaceOffice suiteBackupHoaxLevel (video gaming)InternetworkingWave packetSelectivity (electronic)Computer-assisted translationAreaFood energyIntegrated development environmentData qualityBranch (computer science)Presentation of a groupLecture/ConferenceMeeting/Interview
27:05
Exterior algebraCoefficient of determinationLecture/Conference
27:48
Lecture/Conference
Transcript: English(auto-generated)
00:11
Hi, I'm Daniele. I work for Not on the High Street in London. Not on the High Street is a marketplace for small creative businesses.
00:21
We are in Richmond. Richmond is one of the best parts of London. There are incredible parks and riverside pubs. And it doesn't feel like London at all. It feels like a country village inhabited by ambassadors and rich bankers' wives and tech companies.
00:40
Also Richmond is where the Rolling Stones formed. And some of us care a lot about the Rolling Stones. This is a talk about being on call, carrying a pager, being available 24. If you are a DevOps or a developer, you might already know something about this topic.
01:02
If you are a data engineer, I'm going to make a case that you should care more about this topic. But I will also talk about a TV show. And there is an embarrassingly high number of pictures of my cats. And it's a Friday afternoon talk. So I'm going to warm you up by asking you questions and you're going to raise your hands to answer.
01:24
Who is on call right now? Good. Who was on call in the last month? Who was actually called in the last month? Okay, if you did raise your hand the second time and you didn't raise your hand the third time,
01:47
well, you're living the dream. Money for nothing and chips for free. On the menu for you today. We start with light definition. We move on to some mixed advice on what to do during an actual incident.
02:05
Then we have our prevention special. And then we move a bit with a further view. And we talk about motivations and alerting practices. And in the spirit of total transparencies, total transparency,
02:26
I have a couple of stories to follow. Luckily enough, one was from last week. So some of you already know, but being on call means you have a phone, used to be a pager in the 80s,
02:43
and when it rings, you need to go and fix the system because it's broken. Another way to think about it is basically you have something that is with you at all times and you need to care about it. But secretly it hates you
03:03
and it will demand your attention at the most important times. And when it wants your attention, you have to really give it immediately. But being on call is also about knowledge,
03:20
about being the first person who will act on a certain system in case of a problem. So you get to know an entire system, not only just the parts you did develop. You will also get some rewards because you deserve them.
03:42
Rewards like being woken up in the middle of the night. So last Thursday my phone rang at about 1am and before you get to your computer, you want to make sure you are awake.
04:03
You are fully awake, so make some tea, make some coffee, walk off the sleep a bit. A transition, like going from your sleeping self or your working self,
04:20
if the incident is during office hours, to your incident self. So you don't want to be the guy who is developing on one window and fixing production on the other window. The incident deserves your full attention. The first thing you do is you are going to read the alert that woke you up.
04:46
Really read it. At least a couple of times, possibly more. Once you have read it, you probably know where to look. Which system broke? Where can I find the error logs? Where can I find the monitoring data?
05:03
And you want to gather as much information as you can until you can basically be sure of why did this alert trigger. What did wake me up? And at this point, you probably have enough information to assess the impact.
05:22
What is going to happen because of this problem? Who is not going to be able to do their job? Who is not going to be able to know something they want to know? And be nice. If people are impacted by the problem, inform them.
05:41
Most websites have status pages. For internal systems, you probably have an email or a chat tool. And there is a question you might find yourself asking a lot, which is why?
06:02
What is the real deep cause of this problem? And this is not the right time to answer it. If you are going to dive into with your developer mind, try to find out the real root cause of the problem,
06:21
you are going to spend a possibly very long amount of time. And you cannot quantify it. So don't do it. It's not productive at this time. And as you find out information, as you start acting on the system, log what you are doing.
06:42
So this is on a chat app, it's Slack, but you can also just open a blanket email to your team and start typing out what is happening. This is invaluable information, especially if you are preparing at work on being on call.
07:04
And once you have enough information, you probably can come up with one action or a few actions that will limit the impact as much as possible and are safe.
07:20
If we scroll back, and you probably squint a bit, because I'm not sure if it's readable, you are going to find out that I did not follow my own advice, and last week I ended up forking a library at 4 a.m. and trying to patch it.
07:41
It did not solve absolutely anything, and that's because instead of focusing on taking the smallest piece of action, I actually started asking myself why. And that was not useful at the time. The action I should have taken, which we only took in the morning, was just to skip that data integration step.
08:01
True, we wouldn't have known on the day after what the sources of our traffic were, but at least we would have had all the rest of the data in time. Once you have taken enough steps to limit the impact,
08:23
again, be nice, inform the people who are impacted, and then get back to sleep, because you want to be fresh on the next day. Usually, you wake up the day after, and your first thought in your mind is,
08:41
this is all pretty stressful, I don't want it to happen ever again, and at this point you can really ask yourself, why this error occurred, and the best way to do it is an RCA. There is best practices and extensive literature on RCAs,
09:05
so I'm not going to dive too deep, just one slide. You want to put your detective hat on, gather all the information on what actually happened during the incident and before the incident,
09:23
find the root causes, and be sure to leave enough time at the end to decide on some actions that will mitigate those root causes. And it's very easy, in this case, to try to blame someone. Don't do it. So I'm going to tell you a story.
09:43
It's about a nurse in a childcare hospital. She gave the wrong drug to a little child, and the child almost died. So an inquiry was opened, and there was a proposal to fire the nurse,
10:02
but then the commission on the inquiry dived a bit deeper, and they found that the drugs she should have administered and the drugs she actually administered were one next to each other in the same cabinet, and they had similar labels, and they also found out that the nurse had been working 10 hours straight,
10:23
and there was nobody to double-check what medicines, what drugs she was administering. So don't allow yourself to focus on the fault of one person. Always look at the context.
10:43
And from an RCA, you usually get some useful lessons for the future. You probably are already familiar with this. Please be really careful where your systems talk to a third party
11:00
because communication is more scarce and more easily ignored, and watch out for point of friction in internal communication as well. So the root cause for last week's failure was that the GA reporting API has a time-on-site field,
11:20
and they renamed it to session duration, and the old name was deprecated in 2014, but they actually started enforcing the deprecation and failing on API calls last week. Another insight is to really care about your error messages.
11:40
Keep them up to date. Make sure they include everything that can help you during an incident, so checklists, lesson learned, encouragement. So we took three actions. We fixed the root cause of the problem. We renamed the time-on-site to session duration. We scheduled some time to go through all the GA fields that we are using
12:02
and check if any other of them is deprecated. But we also included in the alert message a specific suggestion not to do what I did. So just skip the step. Don't try to dive into causes too much. So let's take a step back to 2015,
12:26
and let's take a broader view. Downton Abbey is a TV show about a wealthy aristocrat British family in the first half of the 20th century.
12:41
It is wonderfully acted. The sceneries, the costumes, the settings are amazing, and it's really on target for Not on the High Street. It's very British. It's so much on target that when we placed a TV advert during the first episode of the season on a Sunday evening in October 2015,
13:01
we had so much sales and so much additional traffic that the site went down. It recovered, and then it went down again when the same advert aired on the plus one channel. On one side, I was sort of lucky because I was not directly involved
13:21
with the consumer website, but on the other side, the situation on the data infrastructure which I looked after was a lot worse. So we had basically no data since Saturday morning because the replica of the production database we used to read from was offline.
13:41
We were in the process of migrating between hosting providers, and there hadn't been enough communication to turn on when the replication between environments will stop, and the only person who could really fix all of this mess was our databases and networking expert, DevOps,
14:05
and he was on a plane back from Russia. As soon as he landed late Sunday evening, his phone rang so many times that by the time he got home, his battery had drained down. Why did we care?
14:20
Two incidents at the same time, one on the consumer website, one on the data infrastructure that we would have used to evaluate the impact of the consumer website incident, and it was a Sunday evening, so the next day is a Monday.
14:43
On Mondays, our data infrastructure sees the most usage because it's both the busiest trading day and also the day where we plan for the week ahead. I do not like Mondays. And this was not a normal Monday.
15:04
At the time, we were a lot more inexperienced than we are today, and we learned a lot from these events. We learned as an organization, we learned as a team, and we changed. So let's talk a bit about the changes we made.
15:22
How do you answer this question? Well, you look at consequences of an error. Who is affected by the service or the data being unavailable or wrong? Do they depend on your service? How much will it cost?
15:41
How much time can they wait for the information? In 2015, we were realizing that our coworkers and our colleagues were increasingly dependent on our data infrastructure, especially for decision-making.
16:04
So if you have a public-facing service or a public-facing website, you probably want to consider some kind of on-call policy because you cannot control how much the external people depend on your service, and you might also have a contract in place
16:22
or your revenue might depend on the external service. In 2016, we started offering to our partners, the people who sell not on the high street, access to a rich dashboard with sales figures and product performance, and at that point, we didn't have any space
16:41
to roll back the on-call policy. But even if you just have an internal service, you should consider on-call because you want your coworkers to spend less time and worry less about checking and double-checking
17:03
that the services are available, and if they spend less time doing that, they will get benefit in their daily work. If you take a step back to Downton Abbey and you sort of know the characters,
17:22
it's not about just keeping them happy. You also want Daisy, the assistant cook, to be the best she can be at her job. And you might pay this increased interest with a little less control on your priorities
17:42
and a little less agility as you need to react to incidents, but in the end, it will be worth it because you are enabling others to rely on your tool, and your stability will enable their success
18:01
as they build more and more on top of your data and your tools, and you will be surprised by the brilliant creative ways in which they can use the service you provide, enabling others. Nothing else matters.
18:28
So it's worth it. We decided it's worth it. How do we make it work? What did we do in the days, in the weeks, in the months after the Downton Abbey debacle to make sure that we could fix problems in time?
18:45
Usually, the very first basic is getting an email when a certain program fails. The real basic, and that's what we had at the time.
19:00
Then you can build on this email. You can attach certain tools that will phone and wake you up, and you can even do it yourself with Puelo if you want, and then you can also send lower priority alerts and messages to your chat or to your internal communications
19:22
so that you have a timeline of there was this low priority alert, then there was this high priority alert, and this is what happened, and it's all in one place. In this phase, you also want to be making sure that the person answering the incident, responding to the incident,
19:44
is able to do it. So make sure your logs are accessible. Make sure there is documentation in place. Consider training people with fake emergencies and fake incidents. The next step up in the chain is moving from gathering information
20:05
just when bad things happen to gathering information all the time. So you can start with very basic information, CPU usage, RAM usage, disk usage, and then you can move up and take a broader view.
20:24
How many web pages are we serving? How many jobs are running? How much data are we moving? And then you can move even higher. How many customers are we serving? How many orders have been placed? And at this point, you can plug your alerting system
20:45
on top of your monitoring system. Rather than just getting an alert and getting paged when there is a problem, you can say, okay, I have all CPUs at 100% for 10 minutes.
21:05
Maybe it's time for an alert, or I have only 8MB left on my hard drive. Maybe it's time for an alert. And even higher step up the chain is looking at your monitoring and your business data
21:24
and monitor that data itself. So you're looking at questions like is 20,000 customers on the site normal for a Sunday evening? Have we received the data we expected from Google Analytics? Do we have a high rate of traffic
21:45
that doesn't have a Google Analytics identified source? And this works really well because it's basically an alerting system for both your business and your systems.
22:02
There is a lot of literature on data quality, sort of tainted by association with some well-known big software vendors. But if you discard the big software vendors, these concepts are actually general and they don't depend on specific technologies.
22:24
So now we have lots of checks and alerts. Some require immediate attention. Some require attention on the next day. Some require attention on the next working day. And you maybe start ignoring some of them.
22:40
Don't get comfortably numb. Read each alert. Make sure the team reads each alert. Respond to each alert. And also examine if that alert was useful. Can you improve it? Should you silence it? Should you measure something else?
23:03
And then classify alerts. Classify them by system, by kind of problem, by business area, by priority. Ideally every new feature or every new bug is monitoring and alerting attached. And over time you can use information
23:22
you gather this way to guide your decisions. Guide your technical decisions. Guide your product decisions. So I have a very opinionated selection of resources.
23:42
A blog from Julia Evans. A conversation on Twitter with charity majors. The nurse stories stolen from this course on business ethics. I cannot recommend this course enough.
24:00
Even if it doesn't almost cover on call, it's a really good course. And the last one is a 2013 book on data quality, which still holds well enough. I just want to say thank to all the developers and engineers
24:23
and developers who have been on call for years. And make the internet work. Thank you for your presentation.
24:41
And now for questions and answers. You've mentioned training people with fake emergencies and stuff like that. How do you simulate that? Do you actually break something on a staging environment?
25:03
Or what do you do? I actually break production. On purpose during office hours. That's how I do it. I'm not telling you to do it, but that's actually how we do it in my team.
25:29
When you are continuing on this, when you break on production, do you have also another native backup system? It depends on the breakage. If we are causing a breakage on purpose,
25:43
we tend not to do something that will actually go to the customers. So maybe we put a wrong string for the connection database, and then we make it fail before it deploys. We tend not to do anything that will actually impact people.
26:05
I also look after a lot of ETLs. So you can make an ETL fail in the middle of the day, and the data will still be the same data you gather at the beginning of the day. So that's another way to do it. That's mostly how we do it, actually.
26:26
Thank you. Any other questions? It's kind of unrelated, but are those your cats?
26:41
Yes, yes. The grey one is Estia, and the white one is Filo. Thanks, and I know what you feel when they call you. Which kind of software do you use for monitoring your system or your data?
27:07
So this layer is Datadog. They have a booth just right outside.
27:22
For this layer, we use a data democratization tool called Redash. It's a wonderful tool, and I strongly encourage you to try Redash. It's in Python, by the way.
27:41
Obviously, there are alternatives. There are as much as alternatives as you can think of. OK, do we have another question? If not, we can thank our speaker.