Slightly Less Painful Time Zones
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 19 | |
| Number of Parts | 94 | |
| Author | ||
| License | CC Attribution - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/30702 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
RailsConf 201519 / 94
1
4
7
8
9
10
11
13
14
16
17
19
21
24
25
29
30
33
34
35
36
37
39
40
42
47
48
49
50
51
53
54
55
58
59
61
62
64
65
66
67
68
70
71
77
79
81
82
85
86
88
92
94
00:00
Software testingMultiplication signForm (programming)Group actionMereologyElement (mathematics)Lattice (order)Right angleKernel (computing)Game theoryWordCategory of beingArithmetic meanAreaExecution unitDisk read-and-write headCurveEndliche ModelltheorieProcess (computing)Data managementPlanningMedical imagingLevel (video gaming)Food energyUniverse (mathematics)Student's t-testWeightResultantOperator (mathematics)Numerical analysisInsertion lossTask (computing)Social classSimilarity (geometry)Special unitary groupWell-formed formulaPattern languageUniform resource locatorSet theoryPhase transitionTheoryBit rateSummierbarkeitInheritance (object-oriented programming)FreezingContext awarenessExtension (kinesiology)Line (geometry)OntologyLimit (category theory)Physical systemMetropolitan area networkCASE <Informatik>View (database)DemosceneIntegrated development environmentDomain nameService (economics)Machine visionEvent horizonBackupTraffic reportingServer (computing)CodeEmailMathematicsCountingScheduling (computing)Daylight saving timeFrame problemTerm (mathematics)Different (Kate Ryan album)Ocean currentScripting languageTimestampFlow separationOrder (biology)Client (computing)AdditionChecklistMusical ensembleSubsetElectronic mailing listVotingBitInformationTime zoneSlide ruleOnline helpRevision controlFamilyQuicksortWrapper (data mining)Pairwise comparisonPoint (geometry)LogicInternational Date LineBookmark (World Wide Web)Rule of inferenceVideo gameProgrammer (hardware)Source codeDatabaseReal numberData conversionBlogVortexObject (grammar)Test-driven developmentSurfaceQueue (abstract data type)ParsingSoftware developerTime travelLibrary (computing)4 (number)Vector potentialConsistencySampling (statistics)Uniqueness quantificationVideo game consoleState of matterDefault (computer science)Electronic signatureError messageAliasingComputer configurationComputer fontFlagMultilaterationSoftware bugComputer animation
Transcript: English(auto-generated)
00:17
I am here today to talk about a time when I looked deep into the time vortex and thought
00:28
about it. Is the tail of failure also a prediction? So I'm going to tell you about the mistakes that I made so that you can make different mistakes when it comes to time surfaces.
00:41
So here's my plan for today. I'll talk a little bit about why time is hard for starters to deal with. I'll go over the problem I was trying to solve and how I thought I would try to solve it. And then I'll dissect what the code for that would look like in my multiple attempts at a solution. So show of hands, who's already had to deal with
01:01
time zones? Cool, so we're a long time here, that's cool. And what about people that are just sort of learning through osmosis that kind of look like they're a dragon involved? Okay, cool. So just so that we're kind of on the same page, I wanted to pull
01:27
out a couple fun examples of why it is that dealing with time zones can be pretty easy. So here's one, in that it turns out Argentina didn't use daylight savings at all.
01:41
Yeah, but then for a couple years the government was like, oh let's experiment so that we can try to save energy maybe. And so for each year that they were doing daylight savings, the government would set the start and end dates for it. Which means there wasn't a fixed annual schedule that you could program some logic for, it was completely arbitrary.
02:10
Here's another fun one. I found this in the documentation for the active support extensions actually. So in October 1582, it turns out that days 5 to 14 just don't
02:21
exist entirely due to when there was the switch from the Julian to the Gregorian calendar. Okay, now that we know that historical data. And the funny story I write about this is that when it happened, people actually thought that the church was literally stealing
02:41
days off of their life. They thought that they were going to end up dying 9 days sooner. So notice that's been a couple of really great blog posts on the many, many false assumptions that programmers make about time. Some of my favorites are that even though there
03:01
are usually 24 hours in a day, there aren't just 24 time zones. Because the time zones aren't just an hour apart. They can be 30 minutes apart, 45 minutes apart, all sorts of scary intervals. And another great point is that time doesn't always go forwards. If you want to hear more about that, you know, it really is this big ball
03:28
of wiggly wobbly. Let me speak to you our first lesson, which is to trust nothing, including our teaching habits. I don't mean to if there's any art you can use
03:43
whatever you think I'm able to have with you. If you want more proof, here's another example from a Rails console. We do a simple comparison one month to 30 days and we find that yes, those two are equivalent. And then we try to get, you know, today's date or the
04:01
date at the time I was writing this slide together and that was, you know, the same April 9th, fair enough. Let's do some math. We knew one month ago from today and we get March 9th, 8 minus 30 days, which remember, these two are equivalent.
04:21
The answer is in fact not the same. So just remember trust nothing, including Argentina and including math. Trust it either. So I'm going to talk a little bit about the problem.
04:40
At work we have a feature to send weekly email reports to our customers. Because as we like to say, Mondays you need a little bit of help. We have a client job that computes the weekly email jobs for each account that then run in order to build and send those email reports. And so in this way we have a separation between scheduling something
05:03
versus executing it or what you're doing versus when you're doing it. Now we would run this client job that takes everything off on Monday mornings at 10 a.m. Pacific time. And the reasons for this are historical. New Relic is a company that started in the Pacific time zone.
05:24
And so this is the kind of thing that it makes sense when you're small and you don't have that wide spread of customers. Also at New Relic we have a culture of people not working on the weekends. And so it's nice to have something like this on Monday morning in case there are any problems.
05:40
So people can look at that and not have to be checked. But starting at that 10 a.m. for us in the Pacific time, we're already leveled into the current business week for much earlier time zones such as Tokyo where the entirety of Monday has already passed for this report that we're supposed to get on Mondays.
06:04
In addition, New Relic has grown to have many more accounts than when it first started. And with more and more emails sent out, it was even longer and longer to finish which led to even more customers receiving their Monday weekly report on Tuesday in their own time instead.
06:22
With weekly summaries that were offset from what they would expect. So here's our proposed solution. It's a Monday morning report. People should get it by their Monday morning, not ours. So here's a sort of sample of some potential accounts of time zones and what it would mean for us in the Pacific time and when those reports should go out.
06:45
Anyway, as we set it up at 1 a.m. to avoid any weirdness, eventually leaving at the top of the hour. So this is, again, the setup for the whole thing that we have here. We set them up to run at 1 a.m. local time. And since some of those earlier jobs would have to be running on our Sunday,
07:02
we'll kick off the whole process a lot earlier on Saturday. Another nice bonus here, which is that the reports will get sent out spread over a day rather than trying to queue them up all at once. And so we can smooth out that big spike we used to see on our background here every Monday morning.
07:28
Here's my first attack of that. That is, what date is the next Monday on? And in this case, playing around with the console, I saw here we have a date library.
07:41
We have today's date. And we have this class DateTime as well, which is this method of parse. So we can feed it a day of the week in a time, and it creates a date object out of it. So here we've gotten Monday 1 a.m. So now that I have that, all I need to do is convert it into the world we had in the zone for that account.
08:05
So I'm going to do a quick overview of some relevant time and date classes. First, real estate has used a gem called TV info to be able to convert times between different time zones with daylight saving.
08:21
The built-in data source for that is the IAEA time zone database. And it's everything and every so often to reflect changes to time zone boundaries, and change to UTC offset or daylight saving rules. And so you might think that this is pretty static, but it updates quite regularly, actually.
08:40
This is 22 updates in the last three years. And so you need newer versions of TV info to keep up with these changes. Incidentally, ActiveRecord begins that TV info version, so you can't actually update it without updating Rails, unfortunately. You know, if you're in this situation, we're working on it, but we'll see how it goes.
09:04
Anyway, there's also this other useful class of actors support time zone, which is a wrapper around the TV info time zone. And among other things, it lets us limit the set of time zones from TV info to a quote, meaningful subset of 146 out of the many, many more that are available.
09:25
And so if you compare this map of the 24 bands of time you'd expect locally with this list of very tiny fonts of all the time zone names you might have. And so speaking of which, another thing that you get out of this is friendlier time zone names,
09:41
so that instead of having something like America slash New York, you can call a server tag. And so if you want to play around with this in your Rails console to try to get some of those kinds of names, you can run something like this, where I'm just mapping and collecting those values and filtering them for a specific offset.
10:01
So here is the list of time zones with the plus 12 offsets, and these are very early time zones, often while in New Zealand. And then some very late time zones, minus 11, like International, 8, 9, West, 4, 5, and so on. You guys have noticed when you play around, there actually isn't a time zone listed in there
10:21
with UTC minus 12 as the offset. And this is because that time zone covers the Baker and Howlett Islands, which are uninhabited, and so they're not considered for the meaningful subset of time zones. Anyway, back to our question about how do I get that date time into the local time zone.
10:42
Well, that actress important for the time zone class provides us with this method, local to UTC, which given a time zone, it'll convert it into a useful. So this is what the method ended up looking like for this first attack.
11:02
We have this method signature that takes in the account time zones. We use that date time dot parse to get us the next Monday at 1 am. If for whatever reason that account has a time zone set, we're going to use a default of specific time, which is the behavior that we've had before this change in that case.
11:22
And then we'll just go ahead and use that local to UTC method. Here's your part. So, yes, of course we have testing related to time stuff. It's useful to be able to freeze time or travel forward if you want to have a consistent time frame in these tests.
11:43
Basically, it lets you spoof the system time. And time copy is a gem that you can use with. Rails 4.1, I think, actually has some time travel stuff already built in. But, again, we're not quite on to that. So your test will generally follow this kind of setup, where you freeze time at a specific time, and then you'll have your tests.
12:05
You also have to make sure to have this tear down in there after your tests are done, because otherwise, when you run them, you'll get stuff like this. Negative time here. This is what we're basically able to do with that.
12:21
Anyway, the tests then are set up so that I wrote something like this, where it was nice to have those daytime dot parts in there so that it makes the day and time we're looking at pretty easily readable for the next developer that's in there. And I just had a test here for a time zone west of UTC, mountain time, saying, you know, this is the exact same time I expect to get out.
12:49
So that's one time zone on the west, and then I also picked a time zone on the east of UTC, Bangkok time. Everything passes. Awesome.
13:02
We also talked a little bit about some scheduling changes. So just as a reminder, this is the setup that we've had before, where everything is just queued up immediately, and the queue job starts off at Monday. And this is what we want it to get to, where each one is queued with this local 1 a.m. time.
13:26
So pretty straightforward to changes to start on Saturday morning instead. Just sort of say, like, you know, it's a daily class out there, it's Saturday, go ahead and run this script to start queuing up into time. When that happens, the jobs are now using this .qat versus .q.
13:46
.qat lets you pass it at UTC time stamp of once this time is in the past, you start writing this job. And overall, it just really didn't seem like that could change. I got it reviewed by a couple of people, and everything looked good.
14:02
It was a total scammer. Instead of queuing up a bunch of emails to be sent throughout Sunday and Monday, we started seeing tons of emails getting sent immediately on that Saturday morning instead. And these emails didn't really have useful data in them.
14:21
Actually, I had to run off and find a phone room and get help from a coworker, who really should have been spending half of his family instead, and then we had to go into the servers manually to run this, and it was just the worst. So I'll go over why exactly this could have failed, as well as how I ended up fixing it, but here are a couple things to think about first.
14:42
Another thing here is that I really should have set it up so that there would still be a way to trigger the reports to run immediately, without that time delay schedule, and without having to run that command completely manually. It's really hard to watch out for pitfalls that you don't know are there,
15:00
and as we've already discussed, there are a lot of those for time-related things. And so to encounter the worst-case scenario, you should assume that you're going to mess up something and have a plan for what it is. One of the others that wrote a semi-inspirational quiz that I really like, which is, let's try to make better mistakes tomorrow.
15:22
It's not going to be mistake-free, but we can at least not make the same mistakes that we did. So of course after all of this had happened, I then finally thought of a way that I could have rolled out this change in stages rather than all at once. I could have set it up with some of our internal accounts
15:40
to do a bit of a test run, and have it run through the whole system then to see whether it would work. It ended up being queued up for one-way vocal time for the last Monday. So it's already in the past, and that's why they started running immediately. And the problem that we have here
16:01
is that the code couldn't tell whether it's the closest state or the last state. So we take a look at some of the results in the console here. Date that today, that Thursday, when I was putting the slides together. So let's say that that's that Thursday, all the columns are there. And when we put in that datetime.parse for Monday, when I'm like, yeah, what I wanted
16:22
was this Monday in the future, but what I got was this Monday that had already passed. And so this brings us to lesson number four, which is that today's console results are not in fact tomorrow's console, especially if tomorrow doesn't start my new week.
16:42
Turns out time methods are timed in my own current state when doing testing in the console. And I've just been taking it for granted as part of the environment. I really shouldn't have done that.
17:00
I did. So it turns out that datetime.parse isn't really friends with time-cost freezing over time. So we change the test setup so that we pick a very different date. In this case it's just May 1st. Previously the tests were about April. And then we freeze time then.
17:21
May 1st happens to be a Friday and the date that we want is May 4th, which is the date in the future there. I would assume that datetime.parse would be context-aware if it's somehow going to be giving me the next Monday. So if I pry to the setup for the test then, it's correctly May 1st,
17:40
which is what we had set for datetime. And if we go ahead and run that datetime.parse for Monday at 1 a.m., we in fact get April 6th instead of that May 4th thing. That's pretty strange. And so when we look back at the test setup again,
18:01
essentially the timeout phrase didn't have any effect on the setup for the test at all. And the tests only passed because they happened to run in the right week. Because I had picked dates, I ran out of time, and I picked these dates that confirmed what I already knew from running the console
18:20
and that this happened to work at this moment in time. Then that weekend, I came into work on Monday realizing that I had put a failing test on master affecting everyone that works on that codebase. And it was pretty sad. So that brings us now to lesson number five,
18:41
which is to guard against writing false positive tests. Even though my tests were passing, it was really a fluke that they were passing at all. And I should have been more careful in the tests that I wrote to catch more situations rather than confirming what I already knew to be true. So now I'm going to go through a little bit of the actual solution that I ended up with.
19:02
At this point I was like, oh yeah, a TDD. That's like a thing. So that first step there, where we had these tests before, the very first thing is to instantly use dates that aren't read right now. And it happens to be an actual date. So I'm just picking some random dates. This happens to be some dates in February.
19:23
I also decided to let you know to make it easier for the next person to read it, I'll add in just a quick sanity check that this new date, I think, is in fact a Monday. And so, just so people don't have to go through and look up what day of the week was, February 10th, 2015.
19:45
And this does in fact now start failing and demonstrates that it should be had in a date and about person time. So now I need another way to get that next Monday besides this really simplistic use of date time.
20:01
Going back to that lesson number two, I'm having a backup plan. I realize we might need to run this script after a Saturday if that Saturday job fails. And so this is the idea where if everything goes well, run this script on Saturday morning, all the reports are set up to run in the future still for their local time Mondays
20:20
and the report should cover that most recently. But if something fails though, which we know we need to plan for, we might need to run this script later, like on a Sunday, but it should still pick that same Monday and run the report in transition
20:43
so that if you're writing the script later in the week, it knows to set up the reports for early the following Monday. We have this question of how do we distinguish between what we want to have this week versus next week. And it is kind of confusing to think about
21:01
because there's a bunch of time frames that we're looking for. There's the one that you yourself have to locate it in for me. There's the service time zone, which who even knows, and by the way it's server time, so apparently you can't count on it to be accurate. It's not true there. In fact, the customers time zone
21:20
that we're talking about here, which could be ahead of us or behind of us at the time of writing the script. And so we need to have some kind of logic that, until it's globally, what we're coming up with was, if it's still Monday somewhere anywhere, we'll think of it as this week still, for which we want to run this report.
21:41
And then once it's at least Tuesday everywhere, if we run the script then, it should schedule reports for that next Monday. This is just final lesson six here of figuring out a global consistent logic. More straightforward.
22:01
So this is what the method I'm gonna change you to. We're gonna go ahead and grab them today for one of the last time zones globally that international dateline investments are. First thing, you know, we're gonna do some simple math to figure out the date is until the next Monday day of the month. And then we're gonna, as before,
22:21
you know, get either using the account time zone or a default of a specific time. And then we're gonna use this formatted offset method. This returns the offset of the time zone as a formatted string, so like this plus zero seven. And now that we have all those pieces, we can feed that into the dot person get
22:41
with a specific date that we know is the next Monday, the time, one oh one a.m., which is pinned that it's gonna be the same for all of the accounts, regardless of the date and the time. And then that, that number of hours offset for the correct time zone. And now finally, we get those original two tests.
23:01
After we've learned anything, it's not enough coverage there, and there are some other scenarios we need to test out. So one situation that is very important for us still is whether we're inside or outside U.S. dateline saving time. There's plenty of other education, but for us, this was the beginning.
23:21
When we do that, we do in fact find that there's one of the tests now failing here. And if you look closely, we have, of course, an off by one error. And the reason for this is that that formatted offset method alone doesn't care about dateline saving at all. It has no context of the current date. And so you need to add in this dot now
23:42
in order to get it to take the date. And once we've done that, this is when we've covered this normal case of when we're running the script to set up the unique number of scenarios,
24:00
like when we're running the script for the most recent past week, and it's gotten a bit late. We haven't. And so to set up these examples, I'm going to use a really late time zone, American Samoa, and a really early one, well in June and January. And in this case, what's going on is that
24:21
it's already Monday in the customer count before the report even runs. But it's Sunday where we are. So that report should have really already run and got the schedule with the date that's in. So we're running it on June 1st, our time, and it should parse to the 2nd of June.
24:44
And unfortunately, that still works. Awesome. And as another test along similar lines, I'd say it's really late Monday globally, but it's still Monday somewhere, so we don't want to skip ahead to the next Monday. It should still stay on that same 2nd of June.
25:01
Unfortunately, this test is awesome. Finally, we need to test the scenario where we happen to miss the vote for that last week. And for whatever reason, we're running the script ahead of time on Thursday instead of Saturday. So with this one, we're picking up a time
25:21
where we know for sure it's Tuesday everywhere in the world because it's already Tuesday in Samoa. And then we're going to check what will happen for this vote in account of that. And it should now be, rather than the 2nd of June, it should be the 9th of June. It should still always be.
25:43
And this method now works. So if we look at the summary of the tests, before, when I initially implemented this, there were only these two pretty simplistic tests in here. And now we have much more detailed coverage of all the different scenarios of what we expect to happen here, which is also really useful for testing.
26:05
So for me, I came up with this bit of a checklist for different tests to write in the situation that we're making, these time zone-related changes. I had the two already done, covering time zone, west and east of UTC. I also did the pinning it down exactly
26:21
when you're starting and when you're ending, what the expectations are. And this third bullet point, though, is the first thing that I should have done, and we can do this with a separate round, which is picking those dates inside and outside of the, like, oh, sorry, it's going to end, random dates in the future or the past,
26:41
so just not current weeks that happen to be when I'm submitting this PR anyway. Just picking the dates. But also, in addition to that, I think some dates that are inside or outside of daylight saving. And for greater clarity, I think that added check of making sure that the dates being used as a test
27:02
on our expected day of the week is probably pretty useful for future readers of the code, as well as the scenario that we decided was pretty important for our backup of this test to trigger the script on different days of the week here. And finally, I didn't get around to doing this, especially in terms of
27:21
I thought I was prepping this talk, but I really should also have a test for what happens on the actual dates of daylight saving transitions, and so it's possible if I write this test that I would discover some thoughts in my code still of reports getting duplicated or skipped over potentially.
27:42
And so finally, we have a couple of those scheduling classes that we have as well. Reminder, that must have a backup plan, and so I set it up so that there could still be a way to have the jobs get queued up much of the same way that they were performed, using that .acue versus .acue
28:00
act with a time in case there was some undiscovered bug related to trying to calculate that time that we were going to be doing that. So just with this .acue later flag I'd be able to trigger one pathway. And also that lesson of doing an internal test run, the way that I set that up was I have a couple
28:21
test accounts in there, and I set those test accounts to be at opposite ends time zones, sample account three and ten. And I generated the reports to get sent to myself with that new crewed pathway option with that .acue later flag. And when I emailed it to myself, I used this plus email test alias at the end, just so that
28:42
when I received that I knew where I was getting it from. Put that in there to let it bake for a couple of weekends before actually building this change out to production. And when I did that I emailed my e-mails at Pacific time and I did in fact receive these emails at around one in the morning local time
29:01
for those accounts, 4-2 am and 1-0-7 pm Pacific time due to the offset between those accounts times and the final Pacific time. So finally as a summary of all the lessons we have here, lesson number one, remember trust nothing,
29:20
including Argentina. Do an internal test run if at all possible so that if it fails it's not quite so colloquially visible. Remember too as you're testing that today's consult results are not in fact tomorrow's consult results. Do what you can to guard against writing false positive tests.
29:42
Particularly the checklist that helps them out, at least for some of those situations. And then finally figure out a consistent global logic drill, go a long way instead of clarifying what exactly you're trying to do. Thank you.