Slightly Less Painful Time Zones
Wu, Katherine
For developers, there are two things that are certain for time zones: you can’t avoid having to deal with them, and you will screw them up at some point. There are, however, some ways to mitigate the pain. This talk will discuss tactics for avoiding time zone mayhem, using a feature to send out weekly email reports in a customer’s local time zone as a case study. It will cover idiosyncrasies of how time zones are handled in Ruby and Rails, how to write tests to avoid false positives, and advice on how to release time zone-related code changes more safely.
 So my name is Kaye Luke and I have been working as a software engineer at New Relic for about the past year and a half now. I am here today to talk about Amphime and I worked in chief into the Time War text and I thought about it. It is the tale of Sillier who came also a production. So I am going to tell you about the mistakes that I made so that you make different mistakes when it comes to time circumstances. So here is my plan for today. I will talk a little bit about why time is hard for survivors to deal with. I will go over the problem I was trying to solve and how I thought I would try to solve it. So I am going to start with the tool for that in my multiple attempts at a solution. So show of hands. Who has already had to deal with time zones in the codec for gas? Oh, okay. Cool. So one of them is here. That's cool. And what about people that are just sort of having weird and gross voices that kind of looks like they are dragons involved? Okay. So just so that we are kind of on the same page, I wanted to pull out a couple of fun examples of why it is that dealing with time zones can be pretty exciting. So here is one. It turns out Argentina didn't used to do daylight savings at all. So it was a little bit of a mess. But then for a couple of years the government was like, oh, let's experiment so that we can try to save energy, maybe. And so for each year that we were doing daylight savings, the government would set the start and end dates for it. Which means there wasn't a fixed annual schedule that you could program some logic for. It was completely arbitrary. Here's another one. I found this in the documentation for the active support quarters. So in October 1582, it turns out that days 5 to 14 just don't exist entirely. This oddity is due to when there was the switch from the Julian to the Gregorian calendar. Okay. Now that we know that this is a bit of a sense. And the funny story about this is that when it happened, people actually thought that the church was literally stealing days off of their life. They thought they were going to end up dying nine days sooner. 24. So notice that someone had a couple of really great blockbusters on the many, many false assumptions that programmers make about time. Some of my favorites are that even though there are usually 24 hours a day, there aren't just 24 time zones. Because the time zones aren't just an hour apart. They can be 30 minutes apart, 45 minutes apart, all students have three intervals. And another great writing is that time doesn't always go forwards. If you want to hear more about that, I'm going to try to have these blockbusters slide to be up in the capital. All right. So really we can conclude that it really is just this big ball of wiggly, wobbly kind of stuff. And so that brings me to our first lesson, which is to trust nothing, including our GPMs. I don't know if there's any art and community that I would recommend, I'm going to be able to stab you with a knife. Anyhow, if you want more proof, here's another example for a res console. We do a simple comparison, one month to 30 days, and we find that, yes, those two are good ones. And then we try to get, you know, space gave us the data at the time I was writing this slide together. And that was Thursday, April 9th, fair enough. Let's do some math. We do one month ago from today, and we get March 9th, I don't know if that's fair enough. And then if we go ahead and do that date minus 30 days, which remember, that is two hour of privilege, the answer is in fact, not the same. So just to remember, trust nothing, including Argentina, and including math. Trust or not, there. So I'm going to talk a little bit about the problem that I was trying to solve here. I work, we have a future to set weekly email reports to our customers, because as we like to say, Monday is a little bit of help. This is set up with a client job that can use the weekly email jobs for each account that then run in order to build and send those email reports. And so in this way, we have a separation between scheduling something versus executing it, or what you're doing versus when you're doing it. Now, we would run this client job that takes everything off on Monday mornings at 10 am, the second time. And the reasons for this are historical. New Relic is a company that started in the Pacific time zone. And so this is the kind of thing that it makes sense when you're small, and you don't have that wide spread of customers. Also at New Relic, we have a culture of people not working on the weekends. And so it's great to have something like this on Monday morning in case there are any problems. So people can't get that and not have to be checked with right away. However, even when starting at that 10 am for us in the Pacific time, we're already low into the current business peak for much earlier time zones, such as Tokyo, where the entirety of Monday has already passed for this report, which we're supposed to have on Monday. In addition, New Relic has got to have many more accounts than when it first started. And with more and more emails to send out, it was taking longer and longer to finish all of those jobs, which led to even more customers receiving their Monday and weekly report on Tuesday in their own time instead, with weekly summaries that were offset from what they would expect. So here's our proposed solution. It's a Monday morning report. People should get it by their Monday morning, not ours. So here's a sort of sample of some potential accounts and time zones and what it would mean for us in the Pacific time of when those reports should go out. We set it up at 10 am to avoid any year-tests eventually leaving at the top of the hour. So this is again the set up for the holding that we have here. We set them up to run that 10 am local time. And some of those earlier jobs would have to be running on our Sunday. We'll kick off the whole process a lot earlier on Saturday in order to get ahead of our time. There's another nice bonus here, which is that the reports will get sent out spread over a day rather than trying to queue them up all at once. And so we can smooth out that fixed by we used to see on our background here every morning. And we have fewer community alerts there. So here's my first attempt at this all this. The first question I had is what date is the next Monday? And in this case, playing around with the console, we have a date library that connects to each date. And we have this class date time as well, which is this method first. It gets a day of the week and a time and it creates a date on that. So here we've got a Monday, 1 o'clock and a time. So now that I have that, all I need to do is convert it into the local time zone for that account. So I'm going to do a quick overview of some relevant time and date classes that you might end up using in Rails. First, Rails is used in a different time zone with daylight saving and taking it into account. The built-in data source for that is the IAEA time zone database. And it's everything and every so often to reflect changes into time zone boundaries and changes to UTC offset or daylight saving pools. And so you might think that this is pretty static, but it updates quite quickly really actually. This is 22 updates in the last three years. And so you need newer versions of TZ and SO to develop these changes. But it's not like active record to get this back to the old version, so you can't actually update it without updating Rails. Unfortunately. You know, if you're in a situation where we're working on it, we'll see how it goes. Anyway, there's also this other helpful useful fact of actor support time zone, which is a wrapper around what you think is kind of useful. And among other things, it lets us limit the set of time zones from TZ and SO to a quote, meaningful subset of 146 out of the many, many more that are available. And so if you compare this map of the 24 bands of time you'd expect fully with this listed very tiny hunt of all the next domains you might only get access to. And so speaking of which, another thing that you can get out of this is friendlier friends zone names. So that instead of having something like America Slash in yours, you can call these your kind of kids' sets. And so if you want to play around with this in your Rails console to try to get some of those friends of the names, you can run something like this where I just map me and collect those values and filter them for a specific offset. So here is the list of time zones with the plus 12 offsets, and these are very early time zones, often along in New Zealand. And then some very neat kinds of zones minus 11 like international, native, and west, or other things and whatnot. You guys have noticed when you play around that there actually isn't a time zone listed in here with UTC minus 12 as the offset. And this is because that time zone covers the Baker and Howlett islands which are uninhabited. So they are not considered for the meaningful subset of 146. Anyway, time to our question of how do I get that eight times into the local time zone? Well, that accuracy work, a kind of cost, provides us with this method, local to UTC, which given a time zone and a date and time, it will cover anything that is in there for us. So, very useful. So this is what the method ended up looking like for this first attempt here. We have this method that takes in the account time zone, which is the pass-on end. We use that date and time dot course to get us the next Monday, not 1am. If for whatever reason that account will have a time zone set, we're going to use a default of the second time, which is the behavior that we've had before this change in my case. And then we'll just go ahead and use that local to UTC method to get us the UTC method and want this job to end. Pretty straightforward. So, yes, of course, we have to have a test. And so when you do a testing related to time stuff, it's useful to be able to freeze time or travel forward if you want to have a consistent timeframe for these tests. Basically, it lets you smooth the system time. And time cop is the gender that you need this way. Rails 4.1 actually has some time travel stuff already built in. But again, we're not quite on to the first detail stuff. So your test will generally follow this kind of setup, where you freeze time at a specific time, and then you'll have your tests. We also have to make sure to have this tear down in there after your tests are done, because otherwise, when you run them, you'll get stuff like this. Negative time here is important to be able to park your time. And then that will give you a feeling of that. Anyway, the tests then are set up that I wrote, something like this, where it's nice to have those daytime dot parts in there so that it makes the day and time work pretty easily readable for the next developer that's in there. And I just have a test here for the times of Western UTC and mountain time, saying, you know, this is the exact day and time I expect to get out as a result of this. So, fair enough. So that's one time zone on the west, and I also picked a time zone on the east end UTC, a cop time, just to have it in there. And everything passes. Awesome. We also talked a little bit about some scheduling changes that needed to be made. So just as a reminder, this is the setup that we had before where everything is just achieved up immediately, and the job starts off at Monday afternoon. And this is what we want it to get to, where each one is in queue with this local 1-an time and we'll hold it this start off on Sunday. So pretty straightforward to change this to start on Saturday morning instead. It's a daily contract that we're running until it's Saturday. Go ahead and run this script to start to get to the end of the day. And when that happens, the jobs are now using this dot queue at versus dot queue. Link queue at lets you pass it at that UTC time stamp of once this time is in the past. And overall, it just really didn't seem like I could change. I got it reviewed by people and everything looked good. It was a total failure. Instead of curing up a bunch of emails to be sent throughout Sunday and Monday, we started seeing tons of emails getting sent immediately on that Saturday morning set. And these emails didn't really have useful data. I was at a conference at the time actually and I had to run off and find a phone room and get help from a co-worker who really shouldn't have this family instead. And then we had to go into the service manually to run this. And it was just the worst. So I'll go over why exactly this could fail as well as how I ended up fixing it, but here are a couple things to think about first. Lesson in particular is that I really should have set it up so that there would still be a way to trigger the reports to run immediately without that time delay schedule and without having to run that command. It's really hard to watch out for pitfalls that you don't know are there. And as we already discussed, there are a lot of ways for time-related things. And so to encounter the worst-case scenario, you should assume that you're going to mess up something and have a plan for what you'll fall back to. One of the others that was semi-inspirational because I really liked it is let's try to make better mistakes than that. It's not going to be mistake-free, but we can at least not make the same mistakes that we did before. And also, of course, after all of this happened, I then finally thought of a way that I could have rolled out this change in stages rather than all at once. I could have set it up with some of our internal accounts to do a bit of a test run and have it run through the whole system then to see whether it would work. So what happened here? That week's emails ended up being queued up for one-way vocal time for the last Monday, so it was already in the past and that's why they started running immediately. And the problem that we have here is that the group couldn't tell whether it's the closest date or the last date. So we take a look at some of the results in the console here and date that today, it's that Thursday, and we're just going to slide together. So let's say that that's that Thursday on the calendar there, and when we put in that date time dot course for Monday, 1 o'clock a.m., what I wanted was this Monday in the future, but what I thought was this Monday that had already passed. And so this brings us to lesson number four, which is that today's console results are not, in fact, tomorrow's console results, especially if tomorrow is the start of a new week. Turns out time methods are time-sensitive. I was taking for granted my own current date when doing testing during the console, and I had just been taking it for granted as part of the environment. I really shouldn't have done that. But you have tests, you say. Yes, I did. The problem there was it turns out that date time dot course isn't really friends with time-hots, free things, and travel time. So we changed the test setup so that we pick a very different date. In this case, it's just May 1st, previously the test were about April. And then we freeze time then. May 1st happens to be a Friday, and the date that we want is May 4th, which is the date in the future there. I assume that date time dot course would be context-aware, and somehow no to be given in the next Monday. So if I pride into the setup for the test then, it's correctly May 1st, which is probably headset for dot-coc. And if we go ahead and run that date time dot course from Monday, when I went am, we in fact get April 6th instead of that May 4th that we were looking for. So that's pretty strange. And so when it's right, we look back at the test setup again, essentially the time dot freeze didn't have any effect on the setup for the test at all. And the test only began to pass because they happened to run in the right week, because I had 5 days, or right around the time, and I had 5 days that confirmed what I already knew from running the console and that this happened to work at this moment in time. So even after all the mess from that week, I think it's a work on Monday, realizing that I could put a failing test on master, affecting everyone that works on that codebase. And it was pretty sad. So I bring this now to lesson number five, which is to guard against right and false passing of tests. Even though my tests were passing, it was really a fluke that they were passing at all, and I should have been more careful in the test than I wrote to catch more situations, rather than confirming what I already knew to be true. So now I'm going to go through a little bit of the actual solution that I ended up with. At this point I was like, oh yeah, that's like a thing. We can do that. So that first step there, where we have these tests before, very first thing is it's going to use days that are right around now, and have this be in actual. So I'm just taking some random dates, and I have to use them on the dates of February, and then we'll be able to make them all. And I also decided, as you might know, to make it easier for the next person to read it, I'll add in just a quick sanity check that this new data is in fact on Monday. Given what we're trying to do here. And so just so people don't have to go through and look up what day of the week was, February 10th, 2015. And this doesn't, in fact, now start failing and demonstrates that it should be harder to date your job person than to talk, not be different to each other. So now I need another way to get that next Monday, besides this really simplistic, recent date-backed job course. But also thinking back to that lesson number two, having a backup plan. I realized, you know, we might need to run this script after a Saturday, if that Saturday job fails. And so this is the ideal where, if everything goes well, we run this script on Saturday morning. All the reports are set up to run for the future still for their local time on days, and the report should cover that most recently. But if something fails, though, which we know we need to plan for, we might need to run this script later, like on a Sunday, but it should still pick that same Monday and run the report for that past week. And additionally, if you think about it, at some point, though, we're going to have to transition so that if you're running the script later in the week, it knows to set up the reports for early the following Monday. And that's how we do it there. So we have this question of how do we distinguish between when we want to have this week versus next week? And it is going to be confusing to think about because there's a bunch of time frames that we're looking at here. The one that you yourself have to look at it in for me is that specific time. There's the service time zone, which we've been in this, and by the way, it's server time, so apparently you can't count on it to be accurate. In fact, we have the customer's time zone that we're talking about here, which could be ahead of us or behind of us at the time we're running this script. So we need to have some kind of logic that comes up globally. What I ended up coming up with was if it's still Monday somewhere, any here, we'll think of it as this week still, for which we want to run this report. And then once it's at least Tuesday everywhere, if you run the script then it should schedule reports for that next Monday, which is the final lesson six here of figuring out a global consistent logic that comes with a lot of the edge case coding more straightforward. So this is what the method I'm going to change you to. We're going to go ahead and grab them today for one of the last two times, just globally, that international deadline last year. Firstly, we're going to do some simple math to figure out the days until the next Monday. And then we're going to, as before, get the user to the account time zone or a default of this. And then we're going to use this formatted offset method. This returns the offset of the time zone as a formatted strand, so like this plus zero seven. And now that we have all those pieces, we can feed that entity to that person again with a specific date that we know is the next Monday. The time, 1 o'clock and a.m., which is 10, that is going to be the state for all of the accounts regardless of the date of the time. And then that number of hours offset for the current time zone. And now finally, those original two tests passed. When are we learning anything? It's not enough coverage there and there are some other scenarios we need to test out. So one situation that is very important for us still is whether we're inside or outside U.S. state-like saving plan. There's plenty of other educations, but for us this would be a little bit more coverage. And when we do that, we do in fact find that there's one of the tests now failing here. You'll closely, we should have the person go off by one error. And the reason for this is that that formatted offset method alone doesn't care about data-like saving at all. It has no context of the current date. So you need to add in this dot now in order to get it to take the date into account. And once we've done that, those tests are now passing. So essentially we've covered this normal case of when we're running this script to set up, we'll be picking up dots and that's all I'm taking here right now. But we had a couple of scenarios like when we're running this script for the most recent past week and it's gotten a bit late. And it would get a little on-site for better reason. And so to set up these examples, I'm going to use a really late time zone. Americans are on and a really early one. Well, we're doing the early ones. And in this case, what's going on is that it's already Monday in the customer account before the report even runs. But it's Sunday where we are. So that report should really already run and get got scheduled with the date that's in the past for the account. So we're running it on June 1st our time and it should pass to the second of June. And unfortunately that still works. Awesome. And as another test along similar lines, I'll say it's really late Monday, we'll believe it. It's still Monday somewhere. So let's skip ahead to the next Monday. It should still stay on that same side of the June. Unfortunately this test also passes on June 2. Awesome. And then finally, we need to test the scenario where we happen to miss the boat for that last week. And for whatever reason, we're running the script ahead of time on Thursday instead of Saturday. So with this one, we're picking up a time where we know for sure it's Tuesday at River Rollins. It's already Tuesday in Samoa, the last of the zones. And then we're going to check what will happen for this public account that we have scheduled here. And it should now be, rather than the second of June, it should be the ninth of June. And it should still always be in the future. And set it up there. And this method now works. So if we look at the summary of the tests, before when I initially implemented this, there are only these two pretty simplistic tests in here. And now we have much more detailed coverage of all the different scenarios of what we expect to happen here, which is also really useful for tests at the ISEA application. So for me, I came up with this bit of a checklist for different tests to write in the situation in which these time zone related changes. I have initially already done coverage of time zone, west and east of the ISEA. I also did the pinning it down exactly when you're starting and when you're ending to make it easier to see what the expectations of those changes are. And the third bullet point though is the first thing that I should have done, which is picking those dates inside and outside of the, oh, sorry, it's the pen. And then I'm just going to pick the dates in the future or the past. So just not current weeks that happen to be when I'm submitting this PR anyway. Just thinking about the dates at some point later. But also in addition to that, you can pick some dates that are inside or outside of daylight today. And for greater clarity, I think that I had a checkup thinking sure that the dates being used as a hard to act with the expected date of the week. So that's pretty useful for future makers of the code. As well as the scenario that we decided was pretty important for our backup, this test to trigger the strip on different dates of the week here. And finally, I didn't get around to doing this, especially in terms of, I thought it was perfect this top, but I really should also have a test for what happens on the actual dates of daylight transitions. And so it's possible if I write this test that I would discover some months in my code style of shortboards getting duplicated or scheduled or potentially. And so finally we have a couple of the scheduling process changes that we have as well. A writer that must have a backup plan. And so I set it up so that there could still be a way to have the jobs get queued up, much in the same way that they were before using that dot a queue versus dot a queue act with a time in case there was some un-discovered one related to trying to calculate that time that we're going to be doing with that. So just with this queue later flag, I'd be able to trigger one pathway or the other. And also that lesson of doing an internal test run, the way that I set that up was, you know, I have a couple of test accounts in there and I set those test accounts to be at opposite ends times of this sample account three and test. And I generated the reports to get sent to myself with that new cleared pathway option by a queue later flag. And when I emailed it to myself, I used this plus email test that I received that I knew where I was getting from. And put that in there to let it bake for a couple of weekends before actually going this change out to production. And when I did that, I, you know, my email's at the same time and I didn't in fact receive these emails at around one in the morning, both of them from those accounts. For today and 107 p.m. specific time, due to the offset between those accounts times I mean, this is a high-level, this is a high-level. All right. So finally, as a summary of all the lessons we have here, lesson number one, remember, trust nothing, including our 2-9 and including that. Have a backup plan. Do an internal test run, if at all possible, so that if it fails, it's not quite so publicly visible. Remember to guys who are testing that, today's console results are not in fact tomorrow's console results. We can't guard against writing false positive tests, particularly the checklist that helps us get out, these are some of those situations. And finally, figuring out a system that will allow us to go a long way and to verify what it actually is. Thank you.
 So my name is Kaye Luke and I have been working as a software engineer at New Relic for about  the past year and a half now.  I am here today to talk about Amphime and I worked in chief into the Time War text and  I thought about it.  It is the tale of Sillier who came also a production.  So I am going to tell you about the mistakes that I made so that you make different mistakes  when it comes to time circumstances.  So here is my plan for today.  I will talk a little bit about why time is hard for survivors to deal with.  I will go over the problem I was trying to solve and how I thought I would try to solve  it.  So I am going to start with the tool for that in my multiple attempts at a solution.  So show of hands.  Who has already had to deal with time zones in the codec for gas?  Oh, okay.  Cool.  So one of them is here.  That's cool.  And what about people that are just sort of having weird and gross voices that kind of  looks like they are dragons involved?  Okay.  So just so that we are kind of on the same page, I wanted to pull out a couple of fun examples  of why it is that dealing with time zones can be pretty exciting.  So here is one.  It turns out Argentina didn't used to do daylight savings at all.  So it was a little bit of a mess.  But then for a couple of years the government was like, oh, let's experiment so that we  can try to save energy, maybe.  And so for each year that we were doing daylight savings, the government would set the start  and end dates for it.  Which means there wasn't a fixed annual schedule that you could program some logic for.  It was completely arbitrary.  Here's another one.  I found this in the documentation for the active support quarters.  So in October 1582, it turns out that days 5 to 14 just don't exist entirely.  This oddity is due to when there was the switch from the Julian to the Gregorian calendar.  Okay.  Now that we know that this is a bit of a sense.  And the funny story about this is that when it happened, people actually thought that  the church was literally stealing days off of their life.  They thought they were going to end up dying nine days sooner.  24.  So notice that someone had a couple of really great blockbusters on the many, many false assumptions  that programmers make about time.  Some of my favorites are that even though there are usually 24 hours a day, there aren't just  24 time zones.  Because the time zones aren't just an hour apart.  They can be 30 minutes apart, 45 minutes apart, all students have three intervals.  And another great writing is that time doesn't always go forwards.  If you want to hear more about that, I'm going to try to have these blockbusters slide to  be up in the capital.  All right.  So really we can conclude that it really is just this big ball of wiggly, wobbly kind of stuff.  And so that brings me to our first lesson, which is to trust nothing, including our GPMs.  I don't know if there's any art and community that I would recommend, I'm going to be able to stab you with a knife.  Anyhow, if you want more proof, here's another example for a res console.  We do a simple comparison, one month to 30 days, and we find that, yes, those two are good ones.  And then we try to get, you know, space gave us the data at the time I was writing this slide together.  And that was Thursday, April 9th, fair enough.  Let's do some math.  We do one month ago from today, and we get March 9th, I don't know if that's fair enough.  And then if we go ahead and do that date minus 30 days, which remember, that is two hour of  privilege, the answer is in fact, not the same.  So just to remember, trust nothing, including Argentina, and including math.  Trust or not, there.  So I'm going to talk a little bit about the problem that I was trying to solve here.  I work, we have a future to set weekly email reports to our customers, because as we like to say, Monday is a little bit of help.  This is set up with a client job that can use the weekly email jobs for each account that then run in order to build and send those email reports.  And so in this way, we have a separation between scheduling something versus executing it, or what you're doing versus when you're doing it.  Now, we would run this client job that takes everything off on Monday mornings at 10 am, the second time.  And the reasons for this are historical.  New Relic is a company that started in the Pacific time zone.  And so this is the kind of thing that it makes sense when you're small, and you don't have that wide spread of customers.  Also at New Relic, we have a culture of people not working on the weekends.  And so it's great to have something like this on Monday morning in case there are any problems.  So people can't get that and not have to be checked with right away.  However, even when starting at that 10 am for us in the Pacific time, we're already low into the current business peak for much earlier time zones, such as Tokyo, where the entirety of Monday has already passed for this report, which we're supposed to have on Monday.  In addition, New Relic has got to have many more accounts than when it first started.  And with more and more emails to send out, it was taking longer and longer to finish all of those jobs, which led to even more customers receiving their Monday and weekly report on Tuesday in their own time instead, with weekly summaries that were offset from what they would expect.  So here's our proposed solution.  It's a Monday morning report. People should get it by their Monday morning, not ours.  So here's a sort of sample of some potential accounts and time zones and what it would mean for us in the Pacific time of when those reports should go out.  We set it up at 10 am to avoid any year-tests eventually leaving at the top of the hour.  So this is again the set up for the holding that we have here.  We set them up to run that 10 am local time.  And some of those earlier jobs would have to be running on our Sunday. We'll kick off the whole process a lot earlier on Saturday in order to get ahead of our time.  There's another nice bonus here, which is that the reports will get sent out spread over a day rather than trying to queue them up all at once.  And so we can smooth out that fixed by we used to see on our background here every morning.  And we have fewer community alerts there.  So here's my first attempt at this all this.  The first question I had is what date is the next Monday?  And in this case, playing around with the console, we have a date library that connects to each date.  And we have this class date time as well, which is this method first.  It gets a day of the week and a time and it creates a date on that.  So here we've got a Monday, 1 o'clock and a time.  So now that I have that, all I need to do is convert it into the local time zone for that account.  So I'm going to do a quick overview of some relevant time and date classes that you might end up using in Rails.  First, Rails is used in a different time zone with daylight saving and taking it into account.  The built-in data source for that is the IAEA time zone database.  And it's everything and every so often to reflect changes into time zone boundaries and changes to UTC offset or daylight saving pools.  And so you might think that this is pretty static, but it updates quite quickly really actually.  This is 22 updates in the last three years.  And so you need newer versions of TZ and SO to develop these changes.  But it's not like active record to get this back to the old version, so you can't actually update it without updating Rails.  Unfortunately.  You know, if you're in a situation where we're working on it, we'll see how it goes.  Anyway, there's also this other helpful useful fact of actor support time zone, which is a wrapper around what you think is kind of useful.  And among other things, it lets us limit the set of time zones from TZ and SO to a quote, meaningful subset of 146 out of the many, many more that are available.  And so if you compare this map of the 24 bands of time you'd expect fully with this listed very tiny hunt of all the next domains you might only get access to.  And so speaking of which, another thing that you can get out of this is friendlier friends zone names.  So that instead of having something like America Slash in yours, you can call these your kind of kids' sets.  And so if you want to play around with this in your Rails console to try to get some of those friends of the names, you can run something like this where I just map me and collect those values and filter them for a specific offset.  So here is the list of time zones with the plus 12 offsets, and these are very early time zones, often along in New Zealand.  And then some very neat kinds of zones minus 11 like international, native, and west, or other things and whatnot.  You guys have noticed when you play around that there actually isn't a time zone listed in here with UTC minus 12 as the offset.  And this is because that time zone covers the Baker and Howlett islands which are uninhabited.  So they are not considered for the meaningful subset of 146.  Anyway, time to our question of how do I get that eight times into the local time zone?  Well, that accuracy work, a kind of cost, provides us with this method, local to UTC, which given a time zone and a date and time, it will cover anything that is in there for us.  So, very useful.  So this is what the method ended up looking like for this first attempt here.  We have this method that takes in the account time zone, which is the pass-on end.  We use that date and time dot course to get us the next Monday, not 1am.  If for whatever reason that account will have a time zone set, we're going to use a default of the second time, which is the behavior that we've had before this change in my case.  And then we'll just go ahead and use that local to UTC method to get us the UTC method and want this job to end.  Pretty straightforward. So, yes, of course, we have to have a test.  And so when you do a testing related to time stuff, it's useful to be able to freeze time or travel forward if you want to have a consistent timeframe for these tests.  Basically, it lets you smooth the system time.  And time cop is the gender that you need this way.  Rails 4.1 actually has some time travel stuff already built in.  But again, we're not quite on to the first detail stuff.  So your test will generally follow this kind of setup, where you freeze time at a specific time, and then you'll have your tests.  We also have to make sure to have this tear down in there after your tests are done, because otherwise, when you run them, you'll get stuff like this.  Negative time here is important to be able to park your time.  And then that will give you a feeling of that.  Anyway, the tests then are set up that I wrote, something like this, where it's nice to have those daytime dot parts in there so that it makes the day and time work pretty easily readable for the next developer that's in there.  And I just have a test here for the times of Western UTC and mountain time, saying, you know, this is the exact day and time I expect to get out as a result of this.  So, fair enough. So that's one time zone on the west, and I also picked a time zone on the east end UTC, a cop time, just to have it in there.  And everything passes. Awesome.  We also talked a little bit about some scheduling changes that needed to be made.  So just as a reminder, this is the setup that we had before where everything is just achieved up immediately, and the job starts off at Monday afternoon.  And this is what we want it to get to, where each one is in queue with this local 1-an time and we'll hold it this start off on Sunday.  So pretty straightforward to change this to start on Saturday morning instead. It's a daily contract that we're running until it's Saturday. Go ahead and run this script to start to get to the end of the day.  And when that happens, the jobs are now using this dot queue at versus dot queue.  Link queue at lets you pass it at that UTC time stamp of once this time is in the past.  And overall, it just really didn't seem like I could change. I got it reviewed by people and everything looked good.  It was a total failure.  Instead of curing up a bunch of emails to be sent throughout Sunday and Monday, we started seeing tons of emails getting sent immediately on that Saturday morning set.  And these emails didn't really have useful data.  I was at a conference at the time actually and I had to run off and find a phone room and get help from a co-worker who really shouldn't have this family instead.  And then we had to go into the service manually to run this. And it was just the worst.  So I'll go over why exactly this could fail as well as how I ended up fixing it, but here are a couple things to think about first.  Lesson in particular is that I really should have set it up so that there would still be a way to trigger the reports to run immediately without that time delay schedule and without having to run that command.  It's really hard to watch out for pitfalls that you don't know are there. And as we already discussed, there are a lot of ways for time-related things.  And so to encounter the worst-case scenario, you should assume that you're going to mess up something and have a plan for what you'll fall back to.  One of the others that was semi-inspirational because I really liked it is let's try to make better mistakes than that.  It's not going to be mistake-free, but we can at least not make the same mistakes that we did before.  And also, of course, after all of this happened, I then finally thought of a way that I could have rolled out this change in stages rather than all at once.  I could have set it up with some of our internal accounts to do a bit of a test run and have it run through the whole system then to see whether it would work.  So what happened here?  That week's emails ended up being queued up for one-way vocal time for the last Monday, so it was already in the past and that's why they started running immediately.  And the problem that we have here is that the group couldn't tell whether it's the closest date or the last date.  So we take a look at some of the results in the console here and date that today, it's that Thursday, and we're just going to slide together.  So let's say that that's that Thursday on the calendar there, and when we put in that date time dot course for Monday, 1 o'clock a.m., what I wanted was this Monday in the future, but what I thought was this Monday that had already passed.  And so this brings us to lesson number four, which is that today's console results are not, in fact, tomorrow's console results, especially if tomorrow is the start of a new week.  Turns out time methods are time-sensitive.  I was taking for granted my own current date when doing testing during the console, and I had just been taking it for granted as part of the environment.  I really shouldn't have done that.  But you have tests, you say.  Yes, I did.  The problem there was it turns out that date time dot course isn't really friends with time-hots, free things, and travel time.  So we changed the test setup so that we pick a very different date.  In this case, it's just May 1st, previously the test were about April.  And then we freeze time then.  May 1st happens to be a Friday, and the date that we want is May 4th, which is the date in the future there.  I assume that date time dot course would be context-aware, and somehow no to be given in the next Monday.  So if I pride into the setup for the test then, it's correctly May 1st, which is probably headset for dot-coc.  And if we go ahead and run that date time dot course from Monday, when I went am, we in fact get April 6th instead of that May 4th that we were looking for.  So that's pretty strange.  And so when it's right, we look back at the test setup again, essentially the time dot freeze didn't have any effect on the setup for the test at all.  And the test only began to pass because they happened to run in the right week, because I had 5 days, or right around the time, and I had 5 days that  confirmed what I already knew from running the console and that this happened to work at this moment in time.  So even after all the mess from that week, I think it's a work on Monday, realizing that I could put a failing test on master, affecting everyone that works on that codebase.  And it was pretty sad.  So I bring this now to lesson number five, which is to guard against right and false passing of tests.  Even though my tests were passing, it was really a fluke that they were passing at all, and I should have been more careful in the test than I wrote to catch more situations,  rather than confirming what I already knew to be true.  So now I'm going to go through a little bit of the actual solution that I ended up with.  At this point I was like, oh yeah, that's like a thing. We can do that.  So that first step there, where we have these tests before, very first thing is it's going to use days that are right around now, and have this be in actual.  So I'm just taking some random dates, and I have to use them on the dates of February, and then we'll be able to make them all.  And I also decided, as you might know, to make it easier for the next person to read it, I'll add in just a quick sanity check that this new data is in fact on Monday.  Given what we're trying to do here.  And so just so people don't have to go through and look up what day of the week was, February 10th, 2015.  And this doesn't, in fact, now start failing and demonstrates that it should be harder to date your job person than to talk, not be different to each other.  So now I need another way to get that next Monday, besides this really simplistic, recent date-backed job course.  But also thinking back to that lesson number two, having a backup plan.  I realized, you know, we might need to run this script after a Saturday, if that Saturday job fails.  And so this is the ideal where, if everything goes well, we run this script on Saturday morning.  All the reports are set up to run for the future still for their local time on days, and the report should cover that most recently.  But if something fails, though, which we know we need to plan for, we might need to run this script later, like on a Sunday, but it should still pick that same Monday and run the report for that past week.  And additionally, if you think about it, at some point, though, we're going to have to transition so that if you're running the script later in the week, it knows to set up the reports for early the following Monday.  And that's how we do it there.  So we have this question of how do we distinguish between when we want to have this week versus next week?  And it is going to be confusing to think about because there's a bunch of time frames that we're looking at here.  The one that you yourself have to look at it in for me is that specific time.  There's the service time zone, which we've been in this, and by the way, it's server time, so apparently you can't count on it to be accurate.  In fact, we have the customer's time zone that we're talking about here, which could be ahead of us or behind of us at the time we're running this script.  So we need to have some kind of logic that comes up globally.  What I ended up coming up with was if it's still Monday somewhere, any here, we'll think of it as this week still, for which we want to run this report.  And then once it's at least Tuesday everywhere, if you run the script then it should schedule reports for that next Monday,  which is the final lesson six here of figuring out a global consistent logic that comes with a lot of the edge case coding more straightforward.  So this is what the method I'm going to change you to.  We're going to go ahead and grab them today for one of the last two times, just globally, that international deadline last year.  Firstly, we're going to do some simple math to figure out the days until the next Monday.  And then we're going to, as before, get the user to the account time zone or a default of this.  And then we're going to use this formatted offset method. This returns the offset of the time zone as a formatted strand, so like this plus zero seven.  And now that we have all those pieces, we can feed that entity to that person again with a specific date that we know is the next Monday.  The time, 1 o'clock and a.m., which is 10, that is going to be the state for all of the accounts regardless of the date of the time.  And then that number of hours offset for the current time zone.  And now finally, those original two tests passed.  When are we learning anything? It's not enough coverage there and there are some other scenarios we need to test out.  So one situation that is very important for us still is whether we're inside or outside U.S. state-like saving plan.  There's plenty of other educations, but for us this would be a little bit more coverage.  And when we do that, we do in fact find that there's one of the tests now failing here.  You'll closely, we should have the person go off by one error.  And the reason for this is that that formatted offset method alone doesn't care about data-like saving at all.  It has no context of the current date.  So you need to add in this dot now in order to get it to take the date into account.  And once we've done that, those tests are now passing.  So essentially we've covered this normal case of when we're running this script to set up, we'll be picking up dots and that's all I'm taking here right now.  But we had a couple of scenarios like when we're running this script for the most recent past week and it's gotten a bit late.  And it would get a little on-site for better reason.  And so to set up these examples, I'm going to use a really late time zone.  Americans are on and a really early one.  Well, we're doing the early ones.  And in this case, what's going on is that it's already Monday in the customer account before the report even runs.  But it's Sunday where we are.  So that report should really already run and get got scheduled with the date that's in the past for the account.  So we're running it on June 1st our time and it should pass to the second of June.  And unfortunately that still works. Awesome.  And as another test along similar lines, I'll say it's really late Monday, we'll believe it.  It's still Monday somewhere.  So let's skip ahead to the next Monday. It should still stay on that same side of the June.  Unfortunately this test also passes on June 2. Awesome.  And then finally, we need to test the scenario where we happen to miss the boat for that last week.  And for whatever reason, we're running the script ahead of time on Thursday instead of Saturday.  So with this one, we're picking up a time where we know for sure it's Tuesday at River Rollins. It's already Tuesday in Samoa, the last of the zones.  And then we're going to check what will happen for this public account that we have scheduled here.  And it should now be, rather than the second of June, it should be the ninth of June.  And it should still always be in the future.  And set it up there. And this method now works.  So if we look at the summary of the tests, before when I initially implemented this, there are only these two pretty simplistic tests in here.  And now we have much more detailed coverage of all the different scenarios of what we expect to happen here, which is also really useful for tests at the ISEA application.  So for me, I came up with this bit of a checklist for different tests to write in the situation in which these time zone related changes.  I have initially already done coverage of time zone, west and east of the ISEA.  I also did the pinning it down exactly when you're starting and when you're ending to make it easier to see what the expectations of those changes are.  And the third bullet point though is the first thing that I should have done, which is picking those dates inside and outside of the, oh, sorry, it's the pen.  And then I'm just going to pick the dates in the future or the past. So just not current weeks that happen to be when I'm submitting this PR anyway.  Just thinking about the dates at some point later.  But also in addition to that, you can pick some dates that are inside or outside of daylight today.  And for greater clarity, I think that I had a checkup thinking sure that the dates being used as a hard to act with the expected date of the week.  So that's pretty useful for future makers of the code. As well as the scenario that we decided was pretty important for our backup, this test to trigger the strip on different dates of the week here.  And finally, I didn't get around to doing this, especially in terms of, I thought it was perfect this top, but I really should also have a test for what happens on the actual dates of daylight transitions.  And so it's possible if I write this test that I would discover some months in my code style of shortboards getting duplicated or scheduled or potentially.  And so finally we have a couple of the scheduling process changes that we have as well.  A writer that must have a backup plan.  And so I set it up so that there could still be a way to have the jobs get queued up, much in the same way that they were before using that dot a queue versus dot a queue act with a time in case there was some un-discovered one related to trying to calculate that time that we're going to be doing with that.  So just with this queue later flag, I'd be able to trigger one pathway or the other.  And also that lesson of doing an internal test run, the way that I set that up was, you know, I have a couple of test accounts in there and I set those test accounts to be at opposite ends times of this sample account three and test.  And I generated the reports to get sent to myself with that new cleared pathway option by a queue later flag.  And when I emailed it to myself, I used this plus email test that I received that I knew where I was getting from.  And put that in there to let it bake for a couple of weekends before actually going this change out to production.  And when I did that, I, you know, my email's at the same time and I didn't in fact receive these emails at around one in the morning, both of them from those accounts. For today and 107 p.m. specific time, due to the offset between those accounts times I mean, this is a high-level, this is a high-level.  All right. So finally, as a summary of all the lessons we have here, lesson number one, remember, trust nothing, including our 2-9 and including that. Have a backup plan.  Do an internal test run, if at all possible, so that if it fails, it's not quite so publicly visible.  Remember to guys who are testing that, today's console results are not in fact tomorrow's console results.  We can't guard against writing false positive tests, particularly the checklist that helps us get out, these are some of those situations.  And finally, figuring out a system that will allow us to go a long way and to verify what it actually is. Thank you.
Slightly_Less_Painful_Time_Zones_0_47.png Slightly_Less_Painful_Time_Zones_48_155.png Slightly_Less_Painful_Time_Zones_156_299.png Slightly_Less_Painful_Time_Zones_300_2327.png Slightly_Less_Painful_Time_Zones_2328_3119.png Slightly_Less_Painful_Time_Zones_3120_3227.png Slightly_Less_Painful_Time_Zones_3228_3851.png Slightly_Less_Painful_Time_Zones_3852_4223.png Slightly_Less_Painful_Time_Zones_4224_5159.png Slightly_Less_Painful_Time_Zones_5160_5207.png Slightly_Less_Painful_Time_Zones_5208_5315.png Slightly_Less_Painful_Time_Zones_5316_6539.png Slightly_Less_Painful_Time_Zones_6540_6683.png Slightly_Less_Painful_Time_Zones_6684_9719.png Slightly_Less_Painful_Time_Zones_9720_11171.png Slightly_Less_Painful_Time_Zones_11172_13331.png Slightly_Less_Painful_Time_Zones_13332_13547.png Slightly_Less_Painful_Time_Zones_13548_17327.png Slightly_Less_Painful_Time_Zones_17328_17915.png Slightly_Less_Painful_Time_Zones_17916_19919.png Slightly_Less_Painful_Time_Zones_19920_20171.png Slightly_Less_Painful_Time_Zones_20172_21035.png Slightly_Less_Painful_Time_Zones_21036_21467.png Slightly_Less_Painful_Time_Zones_21468_26951.png Slightly_Less_Painful_Time_Zones_26952_27611.png Slightly_Less_Painful_Time_Zones_27612_27839.png Slightly_Less_Painful_Time_Zones_27840_38675.png Slightly_Less_Painful_Time_Zones_38676_41747.png Slightly_Less_Painful_Time_Zones_41748_42347.png Slightly_Less_Painful_Time_Zones_42348_43823.png Slightly_Less_Painful_Time_Zones_43824_45059.png Slightly_Less_Painful_Time_Zones_45060_45143.png Slightly_Less_Painful_Time_Zones_45144_45191.png Slightly_Less_Painful_Time_Zones_45192_45263.png Slightly_Less_Painful_Time_Zones_45264_45359.png Slightly_Less_Painful_Time_Zones_45360_45375.png
So my name is Kaye Luke and I have been working as a software engineer at New Relic for about the past year and a half now. I am here today to talk about Amphime and I worked in chief into the Time War text and I thought about it. It is the tale of Sillier who came also a production. So I am going to tell you about the mistakes that I made so that you make different mistakes when it comes to time circumstances. So here is my plan for today. I will talk a little bit about why time is hard for survivors to deal with. I will go over the problem I was trying to solve and how I thought I would try to solve it. So I am going to start with the tool for that in my multiple attempts at a solution. So show of hands. Who has already had to deal with time zones in the codec for gas? Oh, okay. Cool. So one of them is here. That's cool. And what about people that are just sort of having weird and gross voices that kind of looks like they are dragons involved? Okay. So just so that we are kind of on the same page, I wanted to pull out a couple of fun examples of why it is that dealing with time zones can be pretty exciting. So here is one. It turns out Argentina didn't used to do daylight savings at all. So it was a little bit of a mess. But then for a couple of years the government was like, oh, let's experiment so that we can try to save energy, maybe. And so for each year that we were doing daylight savings, the government would set the start and end dates for it. Which means there wasn't a fixed annual schedule that you could program some logic for. It was completely arbitrary. Here's another one. I found this in the documentation for the active support quarters. So in October 1582, it turns out that days 5 to 14 just don't exist entirely. This oddity is due to when there was the switch from the Julian to the Gregorian calendar. Okay. Now that we know that this is a bit of a sense. And the funny story about this is that when it happened, people actually thought that the church was literally stealing days off of their life. They thought they were going to end up dying nine days sooner. 24. So notice that someone had a couple of really great blockbusters on the many, many false assumptions that programmers make about time. Some of my favorites are that even though there are usually 24 hours a day, there aren't just 24 time zones. Because the time zones aren't just an hour apart. They can be 30 minutes apart, 45 minutes apart, all students have three intervals. And another great writing is that time doesn't always go forwards. If you want to hear more about that, I'm going to try to have these blockbusters slide to be up in the capital. All right. So really we can conclude that it really is just this big ball of wiggly, wobbly kind of stuff. And so that brings me to our first lesson, which is to trust nothing, including our GPMs. I don't know if there's any art and community that I would recommend, I'm going to be able to stab you with a knife. Anyhow, if you want more proof, here's another example for a res console. We do a simple comparison, one month to 30 days, and we find that, yes, those two are good ones. And then we try to get, you know, space gave us the data at the time I was writing this slide together. And that was Thursday, April 9th, fair enough. Let's do some math. We do one month ago from today, and we get March 9th, I don't know if that's fair enough. And then if we go ahead and do that date minus 30 days, which remember, that is two hour of privilege, the answer is in fact, not the same. So just to remember, trust nothing, including Argentina, and including math. Trust or not, there. So I'm going to talk a little bit about the problem that I was trying to solve here. I work, we have a future to set weekly email reports to our customers, because as we like to say, Monday is a little bit of help. This is set up with a client job that can use the weekly email jobs for each account that then run in order to build and send those email reports. And so in this way, we have a separation between scheduling something versus executing it, or what you're doing versus when you're doing it. Now, we would run this client job that takes everything off on Monday mornings at 10 am, the second time. And the reasons for this are historical. New Relic is a company that started in the Pacific time zone. And so this is the kind of thing that it makes sense when you're small, and you don't have that wide spread of customers. Also at New Relic, we have a culture of people not working on the weekends. And so it's great to have something like this on Monday morning in case there are any problems. So people can't get that and not have to be checked with right away. However, even when starting at that 10 am for us in the Pacific time, we're already low into the current business peak for much earlier time zones, such as Tokyo, where the entirety of Monday has already passed for this report, which we're supposed to have on Monday. In addition, New Relic has got to have many more accounts than when it first started. And with more and more emails to send out, it was taking longer and longer to finish all of those jobs, which led to even more customers receiving their Monday and weekly report on Tuesday in their own time instead, with weekly summaries that were offset from what they would expect. So here's our proposed solution. It's a Monday morning report. People should get it by their Monday morning, not ours. So here's a sort of sample of some potential accounts and time zones and what it would mean for us in the Pacific time of when those reports should go out. We set it up at 10 am to avoid any year-tests eventually leaving at the top of the hour. So this is again the set up for the holding that we have here. We set them up to run that 10 am local time. And some of those earlier jobs would have to be running on our Sunday. We'll kick off the whole process a lot earlier on Saturday in order to get ahead of our time. There's another nice bonus here, which is that the reports will get sent out spread over a day rather than trying to queue them up all at once. And so we can smooth out that fixed by we used to see on our background here every morning. And we have fewer community alerts there. So here's my first attempt at this all this. The first question I had is what date is the next Monday? And in this case, playing around with the console, we have a date library that connects to each date. And we have this class date time as well, which is this method first. It gets a day of the week and a time and it creates a date on that. So here we've got a Monday, 1 o'clock and a time. So now that I have that, all I need to do is convert it into the local time zone for that account. So I'm going to do a quick overview of some relevant time and date classes that you might end up using in Rails. First, Rails is used in a different time zone with daylight saving and taking it into account. The built-in data source for that is the IAEA time zone database. And it's everything and every so often to reflect changes into time zone boundaries and changes to UTC offset or daylight saving pools. And so you might think that this is pretty static, but it updates quite quickly really actually. This is 22 updates in the last three years. And so you need newer versions of TZ and SO to develop these changes. But it's not like active record to get this back to the old version, so you can't actually update it without updating Rails. Unfortunately. You know, if you're in a situation where we're working on it, we'll see how it goes. Anyway, there's also this other helpful useful fact of actor support time zone, which is a wrapper around what you think is kind of useful. And among other things, it lets us limit the set of time zones from TZ and SO to a quote, meaningful subset of 146 out of the many, many more that are available. And so if you compare this map of the 24 bands of time you'd expect fully with this listed very tiny hunt of all the next domains you might only get access to. And so speaking of which, another thing that you can get out of this is friendlier friends zone names. So that instead of having something like America Slash in yours, you can call these your kind of kids' sets. And so if you want to play around with this in your Rails console to try to get some of those friends of the names, you can run something like this where I just map me and collect those values and filter them for a specific offset. So here is the list of time zones with the plus 12 offsets, and these are very early time zones, often along in New Zealand. And then some very neat kinds of zones minus 11 like international, native, and west, or other things and whatnot. You guys have noticed when you play around that there actually isn't a time zone listed in here with UTC minus 12 as the offset. And this is because that time zone covers the Baker and Howlett islands which are uninhabited. So they are not considered for the meaningful subset of 146. Anyway, time to our question of how do I get that eight times into the local time zone? Well, that accuracy work, a kind of cost, provides us with this method, local to UTC, which given a time zone and a date and time, it will cover anything that is in there for us. So, very useful. So this is what the method ended up looking like for this first attempt here. We have this method that takes in the account time zone, which is the pass-on end. We use that date and time dot course to get us the next Monday, not 1am. If for whatever reason that account will have a time zone set, we're going to use a default of the second time, which is the behavior that we've had before this change in my case. And then we'll just go ahead and use that local to UTC method to get us the UTC method and want this job to end. Pretty straightforward. So, yes, of course, we have to have a test. And so when you do a testing related to time stuff, it's useful to be able to freeze time or travel forward if you want to have a consistent timeframe for these tests. Basically, it lets you smooth the system time. And time cop is the gender that you need this way. Rails 4.1 actually has some time travel stuff already built in. But again, we're not quite on to the first detail stuff. So your test will generally follow this kind of setup, where you freeze time at a specific time, and then you'll have your tests. We also have to make sure to have this tear down in there after your tests are done, because otherwise, when you run them, you'll get stuff like this. Negative time here is important to be able to park your time. And then that will give you a feeling of that. Anyway, the tests then are set up that I wrote, something like this, where it's nice to have those daytime dot parts in there so that it makes the day and time work pretty easily readable for the next developer that's in there. And I just have a test here for the times of Western UTC and mountain time, saying, you know, this is the exact day and time I expect to get out as a result of this. So, fair enough. So that's one time zone on the west, and I also picked a time zone on the east end UTC, a cop time, just to have it in there. And everything passes. Awesome. We also talked a little bit about some scheduling changes that needed to be made. So just as a reminder, this is the setup that we had before where everything is just achieved up immediately, and the job starts off at Monday afternoon. And this is what we want it to get to, where each one is in queue with this local 1-an time and we'll hold it this start off on Sunday. So pretty straightforward to change this to start on Saturday morning instead. It's a daily contract that we're running until it's Saturday. Go ahead and run this script to start to get to the end of the day. And when that happens, the jobs are now using this dot queue at versus dot queue. Link queue at lets you pass it at that UTC time stamp of once this time is in the past. And overall, it just really didn't seem like I could change. I got it reviewed by people and everything looked good. It was a total failure. Instead of curing up a bunch of emails to be sent throughout Sunday and Monday, we started seeing tons of emails getting sent immediately on that Saturday morning set. And these emails didn't really have useful data. I was at a conference at the time actually and I had to run off and find a phone room and get help from a co-worker who really shouldn't have this family instead. And then we had to go into the service manually to run this. And it was just the worst. So I'll go over why exactly this could fail as well as how I ended up fixing it, but here are a couple things to think about first. Lesson in particular is that I really should have set it up so that there would still be a way to trigger the reports to run immediately without that time delay schedule and without having to run that command. It's really hard to watch out for pitfalls that you don't know are there. And as we already discussed, there are a lot of ways for time-related things. And so to encounter the worst-case scenario, you should assume that you're going to mess up something and have a plan for what you'll fall back to. One of the others that was semi-inspirational because I really liked it is let's try to make better mistakes than that. It's not going to be mistake-free, but we can at least not make the same mistakes that we did before. And also, of course, after all of this happened, I then finally thought of a way that I could have rolled out this change in stages rather than all at once. I could have set it up with some of our internal accounts to do a bit of a test run and have it run through the whole system then to see whether it would work. So what happened here? That week's emails ended up being queued up for one-way vocal time for the last Monday, so it was already in the past and that's why they started running immediately. And the problem that we have here is that the group couldn't tell whether it's the closest date or the last date. So we take a look at some of the results in the console here and date that today, it's that Thursday, and we're just going to slide together. So let's say that that's that Thursday on the calendar there, and when we put in that date time dot course for Monday, 1 o'clock a.m., what I wanted was this Monday in the future, but what I thought was this Monday that had already passed. And so this brings us to lesson number four, which is that today's console results are not, in fact, tomorrow's console results, especially if tomorrow is the start of a new week. Turns out time methods are time-sensitive. I was taking for granted my own current date when doing testing during the console, and I had just been taking it for granted as part of the environment. I really shouldn't have done that. But you have tests, you say. Yes, I did. The problem there was it turns out that date time dot course isn't really friends with time-hots, free things, and travel time. So we changed the test setup so that we pick a very different date. In this case, it's just May 1st, previously the test were about April. And then we freeze time then. May 1st happens to be a Friday, and the date that we want is May 4th, which is the date in the future there. I assume that date time dot course would be context-aware, and somehow no to be given in the next Monday. So if I pride into the setup for the test then, it's correctly May 1st, which is probably headset for dot-coc. And if we go ahead and run that date time dot course from Monday, when I went am, we in fact get April 6th instead of that May 4th that we were looking for. So that's pretty strange. And so when it's right, we look back at the test setup again, essentially the time dot freeze didn't have any effect on the setup for the test at all. And the test only began to pass because they happened to run in the right week, because I had 5 days, or right around the time, and I had 5 days that confirmed what I already knew from running the console and that this happened to work at this moment in time. So even after all the mess from that week, I think it's a work on Monday, realizing that I could put a failing test on master, affecting everyone that works on that codebase. And it was pretty sad. So I bring this now to lesson number five, which is to guard against right and false passing of tests. Even though my tests were passing, it was really a fluke that they were passing at all, and I should have been more careful in the test than I wrote to catch more situations, rather than confirming what I already knew to be true. So now I'm going to go through a little bit of the actual solution that I ended up with. At this point I was like, oh yeah, that's like a thing. We can do that. So that first step there, where we have these tests before, very first thing is it's going to use days that are right around now, and have this be in actual. So I'm just taking some random dates, and I have to use them on the dates of February, and then we'll be able to make them all. And I also decided, as you might know, to make it easier for the next person to read it, I'll add in just a quick sanity check that this new data is in fact on Monday. Given what we're trying to do here. And so just so people don't have to go through and look up what day of the week was, February 10th, 2015. And this doesn't, in fact, now start failing and demonstrates that it should be harder to date your job person than to talk, not be different to each other. So now I need another way to get that next Monday, besides this really simplistic, recent date-backed job course. But also thinking back to that lesson number two, having a backup plan. I realized, you know, we might need to run this script after a Saturday, if that Saturday job fails. And so this is the ideal where, if everything goes well, we run this script on Saturday morning. All the reports are set up to run for the future still for their local time on days, and the report should cover that most recently. But if something fails, though, which we know we need to plan for, we might need to run this script later, like on a Sunday, but it should still pick that same Monday and run the report for that past week. And additionally, if you think about it, at some point, though, we're going to have to transition so that if you're running the script later in the week, it knows to set up the reports for early the following Monday. And that's how we do it there. So we have this question of how do we distinguish between when we want to have this week versus next week? And it is going to be confusing to think about because there's a bunch of time frames that we're looking at here. The one that you yourself have to look at it in for me is that specific time. There's the service time zone, which we've been in this, and by the way, it's server time, so apparently you can't count on it to be accurate. In fact, we have the customer's time zone that we're talking about here, which could be ahead of us or behind of us at the time we're running this script. So we need to have some kind of logic that comes up globally. What I ended up coming up with was if it's still Monday somewhere, any here, we'll think of it as this week still, for which we want to run this report. And then once it's at least Tuesday everywhere, if you run the script then it should schedule reports for that next Monday, which is the final lesson six here of figuring out a global consistent logic that comes with a lot of the edge case coding more straightforward. So this is what the method I'm going to change you to. We're going to go ahead and grab them today for one of the last two times, just globally, that international deadline last year. Firstly, we're going to do some simple math to figure out the days until the next Monday. And then we're going to, as before, get the user to the account time zone or a default of this. And then we're going to use this formatted offset method. This returns the offset of the time zone as a formatted strand, so like this plus zero seven. And now that we have all those pieces, we can feed that entity to that person again with a specific date that we know is the next Monday. The time, 1 o'clock and a.m., which is 10, that is going to be the state for all of the accounts regardless of the date of the time. And then that number of hours offset for the current time zone. And now finally, those original two tests passed. When are we learning anything? It's not enough coverage there and there are some other scenarios we need to test out. So one situation that is very important for us still is whether we're inside or outside U.S. state-like saving plan. There's plenty of other educations, but for us this would be a little bit more coverage. And when we do that, we do in fact find that there's one of the tests now failing here. You'll closely, we should have the person go off by one error. And the reason for this is that that formatted offset method alone doesn't care about data-like saving at all. It has no context of the current date. So you need to add in this dot now in order to get it to take the date into account. And once we've done that, those tests are now passing. So essentially we've covered this normal case of when we're running this script to set up, we'll be picking up dots and that's all I'm taking here right now. But we had a couple of scenarios like when we're running this script for the most recent past week and it's gotten a bit late. And it would get a little on-site for better reason. And so to set up these examples, I'm going to use a really late time zone. Americans are on and a really early one. Well, we're doing the early ones. And in this case, what's going on is that it's already Monday in the customer account before the report even runs. But it's Sunday where we are. So that report should really already run and get got scheduled with the date that's in the past for the account. So we're running it on June 1st our time and it should pass to the second of June. And unfortunately that still works. Awesome. And as another test along similar lines, I'll say it's really late Monday, we'll believe it. It's still Monday somewhere. So let's skip ahead to the next Monday. It should still stay on that same side of the June. Unfortunately this test also passes on June 2. Awesome. And then finally, we need to test the scenario where we happen to miss the boat for that last week. And for whatever reason, we're running the script ahead of time on Thursday instead of Saturday. So with this one, we're picking up a time where we know for sure it's Tuesday at River Rollins. It's already Tuesday in Samoa, the last of the zones. And then we're going to check what will happen for this public account that we have scheduled here. And it should now be, rather than the second of June, it should be the ninth of June. And it should still always be in the future. And set it up there. And this method now works. So if we look at the summary of the tests, before when I initially implemented this, there are only these two pretty simplistic tests in here. And now we have much more detailed coverage of all the different scenarios of what we expect to happen here, which is also really useful for tests at the ISEA application. So for me, I came up with this bit of a checklist for different tests to write in the situation in which these time zone related changes. I have initially already done coverage of time zone, west and east of the ISEA. I also did the pinning it down exactly when you're starting and when you're ending to make it easier to see what the expectations of those changes are. And the third bullet point though is the first thing that I should have done, which is picking those dates inside and outside of the, oh, sorry, it's the pen. And then I'm just going to pick the dates in the future or the past. So just not current weeks that happen to be when I'm submitting this PR anyway. Just thinking about the dates at some point later. But also in addition to that, you can pick some dates that are inside or outside of daylight today. And for greater clarity, I think that I had a checkup thinking sure that the dates being used as a hard to act with the expected date of the week. So that's pretty useful for future makers of the code. As well as the scenario that we decided was pretty important for our backup, this test to trigger the strip on different dates of the week here. And finally, I didn't get around to doing this, especially in terms of, I thought it was perfect this top, but I really should also have a test for what happens on the actual dates of daylight transitions. And so it's possible if I write this test that I would discover some months in my code style of shortboards getting duplicated or scheduled or potentially. And so finally we have a couple of the scheduling process changes that we have as well. A writer that must have a backup plan. And so I set it up so that there could still be a way to have the jobs get queued up, much in the same way that they were before using that dot a queue versus dot a queue act with a time in case there was some un-discovered one related to trying to calculate that time that we're going to be doing with that. So just with this queue later flag, I'd be able to trigger one pathway or the other. And also that lesson of doing an internal test run, the way that I set that up was, you know, I have a couple of test accounts in there and I set those test accounts to be at opposite ends times of this sample account three and test. And I generated the reports to get sent to myself with that new cleared pathway option by a queue later flag. And when I emailed it to myself, I used this plus email test that I received that I knew where I was getting from. And put that in there to let it bake for a couple of weekends before actually going this change out to production. And when I did that, I, you know, my email's at the same time and I didn't in fact receive these emails at around one in the morning, both of them from those accounts. For today and 107 p.m. specific time, due to the offset between those accounts times I mean, this is a high-level, this is a high-level. All right. So finally, as a summary of all the lessons we have here, lesson number one, remember, trust nothing, including our 2-9 and including that. Have a backup plan. Do an internal test run, if at all possible, so that if it fails, it's not quite so publicly visible. Remember to guys who are testing that, today's console results are not in fact tomorrow's console results. We can't guard against writing false positive tests, particularly the checklist that helps us get out, these are some of those situations. And finally, figuring out a system that will allow us to go a long way and to verify what it actually is. Thank you.
already had to deal with time zones in the codec email reports to our customers, because as we And with more and more emails to send out, it was at 10 am to avoid any year-tests eventually set them up to run that 10 am local time. And version, so you can't actually update it without time, it will cover anything that is in there for that time delay schedule and without having to as we already discussed, there are a lot of ways write in the situation in which these time zone the dates at some point later. But also in against writing false positive tests,