The overnight failure
Sogamoso, Sebastian
Close your eyes and imagine what is the worst thing that could happen to you at work. For me, that was charging users thousands of times the amount they should have been charged. In this talk we will find out what caused this particular bug and what we learnt from dealing with the whole situation. Did someone get fired? Did the company survive the bug? Come to the talk and you will learn the answers, but more importantly a thing or two about how to go through tough times at work.
 Okay, so hi everyone. I hope you're having a good time here. I know this can be a hard time of the day. I'm the only team between Yon lunch and the spring two days of talks. Last night was fun. We had an amazing time at Ruby karaoke. So I'll try to make this entertaining for you. So this is actually my first time at RubyConf. It's also my first time in New Orleans. And I have really enjoyed my time here. But as Aaron, I think, just pointed out, so far the conference team seems to have been about death. That's a bit sad. I really don't think Ruby is dead. But just so we are all clear, let's check the internet because that's where you check when you want to find the sort of truth for something. So let's see it. Okay. Yeah, Ruby is not dead yet. Thanks for building the website, Jason. Yeah, we're safe. Okay, so now that we got that of the way, we can continue. So let me introduce myself. My name is Sebastian. I work at a company called Cookepad. I like when people treat me during my talks. So that's my true handle. I came all the way from Columbia. And if not because it's everything you know about Columbia, I think you should definitely go and visit. And have, like, know the real Columbia. And in fact, like, I have the perfect excuse for you to go there. We have a Ruby conference there. Next one is going to be September 2018. We'll be announcing the exact date really soon. So you can follow the conference on Twitter. It's going to be fun. So lately I've been, like, trying to get into stuff different from programming. So I've been taking some cooking lessons to improve my culinary skills. So when I was accepted to give the suck here, I decided that I wanted to prepare some jam alaya because it's something I've always wanted to try. So I went online, looked for a good recipe. Oh, that's it. Okay, it looks weird there. It looked nice, so I went ahead and bought all the ingredients, prepared everything, followed the steps, and there was a final result. As you can see, it was an epic fail. I definitely have a long way to go. But before we continue, I really want to ask everyone a favor. I'm really happy to be here, so I would really love to take a selfie with you. So if you could all, like, raise your arms, make some noise. I'll do this real quick, I promise. Woo! Yeah, thanks! You're great. Awesome. So now let's get a bit more serious. The name of the talk is the overnight failure. And this talk is based on a true story. Something happened to me on a previous show I had. So why do I want to talk about this? Why do I want to talk to tell everyone about a time I really fucked up? Well, there are a few reasons, and I'll share the most important with you. So first of all, we all have broken the internet at some point. We all have screwed up really bad. We all have created bugs that made introduction at some point. And if it hasn't happened to you yet, it will happen, believe me. It's just a matter of time. And also because we don't normally talk about this in public. Like, at conferences, like meetups, whatever, we see people talking about what they learned, how they built a cool thing, how they made a project be successful. Basically, we mostly hear about success stories. And I think people should really talk more about this in public because, I mean, it will make talking about screw ups easier for everyone. And we normally just talk about this with people that are really close to us. We do it in private, to our families, friends, or maybe our colleagues because they're on the same boat with us. And also because by not speaking about failures in public, I think we boost posture syndrome, which is something terrible that a lot of us have suffered. And although sometimes companies talk about this publicly, for example, like in January, good love, sorry, good love, was down for several hours or the mistake that take a WS and a big chunk of the internet with it down for also a few hours. And I can imagine how stressful the situation was. I even remember how, like, their website showed that everything was right because the red icon that they should have been shown was also down because of S3. So it's something that's easy to laugh about. It's fun when we look at other people's experience. But if you really think about it, you can probably feel how stressful that situation was for the people having to deal with it. And how hard it was to keep customers trust and regain it. So although we don't all work at companies that have such a big scale as good love over Amazon, if we create bugs, they will affect the lives of our users. So it doesn't matter if we have millions or thousands of hundreds of users, it will always be stressful. So that's why, like, basically the most important, I mean, that takes me to the most important reason why I wanted to give this talk and talk about my fuck ups. And it's because we learn more from failure than from success. So when we fail, when we, like, when everything goes wrong or something that we didn't expect to go wrong, goes wrong, it's where we learn the most. So I also want to share a little bit about what I learned. So before we move on, and I tell you my story, I want you to take a few seconds to maybe close your eyes if you want and think what's the worst thing that could happen to you at work? Imagine what could be like the apocalypse for you. So, okay, keep that in mind. And I'll tell you what my overnight failure story was. So how did it happen? How did the overnight failure become a thing? So I used to work at a company that had a, like, product that allowed people to carpool. It was a mobile app. And so the mobile app allowed people to use their car to go to work every day. So for example, let's say this is Mary. So Mary had a car. She used it to go to work and back every day. And then the app I helped build allowed Mary to find Anna who had a similar, who didn't have a car and also had a similar commute. So they had, like, a similar route and the app allowed them to exchange money with one another. So when they, like, shared a car to go to work, the app kept the record of the trips they took together and even allowed them to be matched with someone else, like, for example, John, who also had a similar commute. So they could ride share or share their car every day to go to work and the app would keep track of every trip they took. So how the money exchange part worked is that, like, once a week, there was a process that run that basically charged the passengers and then got the money and paid the driver, paid whoever it was. So for example, in this case, Anna. So going a little bit deeper, explain how this worked technically, how it was implemented, so the process that run once a week looked kind of like this. So every week, like, a job was triggered and the job went to the database and checked, like, every trip that was, every trip that users took during the previous week and then for each driver-passenger combination, it put, like, a job on a queue. We could call it like that. So then it created a lot of jobs for each week and each one of these jobs contained, like I said, mentioned, a passenger-driver combination and the total for the trips that the passenger should pay the driver. So after that, what happened is that each job was processed and the passenger was charged using a payment gateway and then the driver was paid and what's there on the circle was basically managed by the payment gateway. It wasn't managed by our system. The payment gateway took care of all of that. So I don't know if you're still with me, so let's do a quick recap. So basically, we had an app that users used to call Carpool every day that took care of the payment process each week. Passengers were charged and drivers were paid for the trips. So now that you kind of understand how it worked, I'm going to tell you exactly what happened. So it was, and given Sunday, it was 6 a.m. where the process was triggered, when the process was triggered, and started usual, the process runs started, like, yeah, the process runs started the marked start of the day that I like to call Black Saturday. And it has nothing to do with, like, big discounts and stuff, like shopping spree, a really fun day. It has to do with a really, really stressful day. So I'll tell you what happened next. So then, let's say you have a user that went really early to the store to buy, like, some excellent bread for her breakfast, for her family, and then when she tried to pay, her card was declined. And she was like, well, this is weird. This shouldn't happen. And this happened, like, around 6.25 a.m. So she went online and investigated what was going on with her credit card, and she saw a lot of charges that were made by the car pulling company I used to work for on that same morning. So that's the second step of her, like, Black Saturday. So that was bad. But then things started to look even worse because, like, more users were affected by this. And it was not just a few users, it was a bunch of users. So the customer care team started to get, like, a huge influx of complaints. It was bad. So let's say that at, like, 6.35, a lot of users in, like, a small amount of time reported, like, a lot of bugs. So that's, I mean, a lot of problems, a lot of charges. So that's where things definitely started to look bad. And so it was 6.34. I was sleeping. I don't know about you, but I'm always sleeping on Saturday at that time. So I was happily sleeping in my bed, and then the phone rang. And it was a call from my boss. And this is roughly how the conversation went. My boss told me, like, hey, sorry to wake you up this early, but there's, like, an issue in production. There's a lot of customers complaining about it. And I was like, okay, sure, like, I'll take a look right away. I was really stressed. I was trying to act cool. So I thought we had ended the call, and I was like, fuck! Fuck! And then my boss said, like, hey, I'm still in the line. It wasn't looking promising. So then, okay, this is how are they, like, black-sided with looking. So it wasn't even 7 AM. It was already crappy. So I started to look into what was going on, and I checked the payment gateway, and I definitely saw a lot of duplicates charges. So the first thing I noticed was that this didn't make any sense. So when I looked at our system, like our billing system, I saw that the queue was still full of jobs. And so the first thing I thought I was doing was, like, to kill that part of the process, so basically stop processing jobs. And that will at least still not create more charges. But then I looked at the queue again, and the queue grew. This was bad. So then I decided to basically stop the whole process so that no more jobs were put into the queue. And then the first thing I thought about was, okay, let's refund all the charges that we shouldn't have made. And this sounded like a really good idea at the moment, but you'll see that later maybe it wasn't. So then another thing I noticed is that a lot of the charges we had already made created a lot of transfers to the drivers, and those were still also happening. So I also had to go and reverse all those transfers and stop the ones that were remaining. So we could say that at, like, 7.28, the problem was kind of contained at this point. So, yeah, the day wasn't still, was still not looking very good. The queue was full. It was full of jobs, had thousands of jobs to be processed. And when I started looking at them, I started noticing a pattern. So there were a lot of duplicated jobs, as you might have expected. A lot of them contained the same user that was a passenger, the same user that was a driver, and the same amount to be charged. So yeah, there were a lot of duplicated ones. So what I noticed is that, like, something, I mean, to me, it was clear that the problem was in two points of the system. The problem was that here on this part of the system, it seems like we were putting every passenger driver pair on the queue thousands of times for some reason. And also that for some reason, when we processed the jobs, we didn't check if we had already charged that passenger-driver combination. We still charged them, no matter what. So after debugging, finding what was going on, finding what was the best to solve this, I wrote some tests that failed and wrote the code to make them pass, deployed it, and it was all kind of fixed, or fixed. But that took a long time. It was a long day. I felt really dumb, really frustrated. And when I was about to deploy the code I wrote, I did the programmer's prayer that Aaron taught to us on RailsConf this year. So please work, please work, please work. Fortunately, it did. So at 10.55 p.m., I started doing the obvious thing, started looking for a new job. Well, that was not true, but I was really frustrated. This is how I felt. Yeah, pretty much. Okay, so this is what had happened. So there were thousands of users affected by the bug. All users were charged a different amount, the worst case being one that was charged over $5,000. And when we should have only charged a user 50, some users were charged up to 500 times. And the worst part was that we maxed out a lot of credit cards and also emptied a lot of savings accounts. So as I had mentioned, I created a refund for every extra charge with it. And the problem was that refunds take a long time. And people didn't have any money on their banks account. So I'll tell you in a few minutes how we dealt with that. I mean, something I was really glad about that is that we were still not on Ruby 3. Because if so many jobs were processing such a little amount of time, I couldn't imagine what would have happened if we were using that. The problem would have been much, much terrible. So as I was mentioning, refunds take up to five business days. So there was unacceptable for a lot of our users. So what we had to do is reach out to all of them and offer them an expedited way to reimburse them, like an expedited option. Some of them were just sending them checks or kind of not be a paypal. We did whatever we could and then figured out what to do with the refund. But this was really stressful. I spent, like, I remember at least the next week just grabbing information about how bad we have fucked up. Yeah, it was really bad. So why did I want to tell you all of this? Like, why is this important? I want to go back to the why. So embarrassing things will happen. It will eventually happen to all of us I've mentioned before. And you might think that tests might save you from this, but they won't. We had tests. That could review might save you from this, but it won't. We had that as well. Southwards built by humans. And as Olivier mentioned on his talk, I think on Wednesday, we're all humans. We make mistakes. So bad things will happen eventually. And we need to make admitting mistakes easy for everyone. It's really important to create a culture where admitting mistakes is fine. And you don't feel like you have to blame others or to make excuses up for a mistake you made. And we need to be trust that we won't be judged. Because, like, I mean, the safer people feel about admitting their mistakes, the more they will learn about them and the more their colleagues will learn about them. The more we as a community will learn about it. So when you're dealing with this sort of situation, like my first recommendation is that you make sure before doing anything that you understand. You really understand what happened. It's really easy to jump to quick conclusions and do, like, whatever you think is the easiest fix or the most quick fix for something. And sometimes we end up making the problem worse as it happened with the refunds. It was a headache, basically, to deal with the people that we sent money in, like, a different type of form. And then we had already, like, done a refund for. It was crazy. It was a lot of, we had to go into, like, a lot of trouble and talk to the payment gateway a lot. By the way, like, I don't know how the payment gateway allows us to do this. Like, at least you have noticed that we were doing something weird. But anyway. My second recommendation would be move really slow. So don't rush. The problem, I mean, there's, you really already screwed up. The problems are ready. People already noticed the problem. So it doesn't mean that the quicker you fix it, I mean, that if you fix it quick, like, some people will notice. That's not the case. So just make sure that you take the time to think what you're really doing, which really goes back to the thing I just mentioned, the first one I just mentioned. Something that I really also learned and that I think a lot of companies do a very good job doing this is document the problem, document what happened, document what the root cause of it was, document how you fixed it, document what you will do to prevent this from happening again. So also document the fix. And document, this is the most important thing, I think, document the lesson learned. Because that's the only way that you will really prevent stuff from happening again. And also, don't do this. Don't get blamed. It's not about who wrote the code where the bug was. That's not important. The important thing is that the process failed, the company failed, the team failed. It's not only the follow the person who wrote the code. That's just part of it. So doing it, blaming is just like not creating a good culture where people will feel safe and will feel like trust to admit their mistakes. And really importantly, you are not your failures. Don't think that because you failed, you did a big screw up. You're the worst developer ever because you're not. Everyone has done it. Every developer you admire, they have screwed up at some point, believe me. And I think chat follow can put it better than me. Like no one will care about your, the bugs you create when you die. So keep in mind that it's so temporary. So as bad as you might feel at the moment that you're dealing with a really bad situation, it will pass. And you will look at it in the past and laugh about it, give talks about it. We all, we're here, I mean, I really want to celebrate our failures because they also make us what we do. So you can come and talk to me, tell me your failures. I've heard a lot and it's a lot of fun. It's kind of therapeutic. And also you can tweet using the hashtag of the conference. It will be great if we share our war stories, our scars. Because as a community, we can help each other and like get us through this like really stressful moments in our careers. We can make it together. And I'll actually let John and P.A. say it better. Oh, too bad you can hear that. So anyway, I think that's all I wanted to say for today. I work for this, oh wait, what happened? Okay, we're back. Okay, I work for this company called Kupad. Go and check it out if you like cooking and sharing your recipes. We're hiring as everyone else. So if you want to work at a company that works, sorry, where you can build like an application that's used by millions of people around the world, come and talk to me and thanks a lot.
 Okay, so hi everyone. I hope you're having a good time here. I know this can be a hard  time of the day. I'm the only team between Yon lunch and the spring two days of talks.  Last night was fun. We had an amazing time at Ruby karaoke. So I'll try to make this  entertaining for you. So this is actually my first time at RubyConf. It's also my first  time in New Orleans. And I have really enjoyed my time here. But as Aaron, I think, just  pointed out, so far the conference team seems to have been about death. That's a bit sad.  I really don't think Ruby is dead. But just so we are all clear, let's check the internet  because that's where you check when you want to find the sort of truth for something. So  let's see it. Okay. Yeah, Ruby is not dead yet. Thanks for building the website, Jason.  Yeah, we're safe. Okay, so now that we got that of the way, we can continue. So let me  introduce myself. My name is Sebastian. I work at a company called Cookepad. I like  when people treat me during my talks. So that's my true handle. I came all the way from Columbia.  And if not because it's everything you know about Columbia, I think you should definitely  go and visit. And have, like, know the real Columbia. And in fact, like, I have the perfect  excuse for you to go there. We have a Ruby conference there. Next one is going to be  September 2018. We'll be announcing the exact date really soon. So you can follow the conference  on Twitter. It's going to be fun. So lately I've been, like, trying to get into stuff  different from programming. So I've been taking some cooking lessons to improve my culinary  skills. So when I was accepted to give the suck here, I decided that I wanted to prepare  some jam alaya because it's something I've always wanted to try. So I went online, looked  for a good recipe. Oh, that's it. Okay, it looks weird there. It looked nice, so I went  ahead and bought all the ingredients, prepared everything, followed the steps, and there was  a final result. As you can see, it was an epic fail. I definitely have a long way to go.  But before we continue, I really want to ask everyone a favor. I'm really happy to be here,  so I would really love to take a selfie with you. So if you could all, like, raise your  arms, make some noise. I'll do this real quick, I promise. Woo! Yeah, thanks! You're great.  Awesome. So now let's get a bit more serious. The name of the talk is the overnight failure.  And this talk is based on a true story. Something happened to me on a previous show I had. So  why do I want to talk about this? Why do I want to talk to tell everyone about a time  I really fucked up? Well, there are a few reasons, and I'll share the most important  with you. So first of all, we all have broken the internet at some point. We all have screwed  up really bad. We all have created bugs that made introduction at some point. And if it  hasn't happened to you yet, it will happen, believe me. It's just a matter of time. And  also because we don't normally talk about this in public. Like, at conferences, like  meetups, whatever, we see people talking about what they learned, how they built a cool thing,  how they made a project be successful. Basically, we mostly hear about success stories. And  I think people should really talk more about this in public because, I mean, it will make  talking about screw ups easier for everyone. And we normally just talk about this with  people that are really close to us. We do it in private, to our families, friends, or  maybe our colleagues because they're on the same boat with us. And also because by not  speaking about failures in public, I think we boost posture syndrome, which is something  terrible that a lot of us have suffered. And although sometimes companies talk about this  publicly, for example, like in January, good love, sorry, good love, was down for several  hours or the mistake that take a WS and a big chunk of the internet with it down for  also a few hours. And I can imagine how stressful the situation was. I even remember how, like,  their website showed that everything was right because the red icon that they should have  been shown was also down because of S3. So it's something that's easy to laugh about.  It's fun when we look at other people's experience. But if you really think about it, you can  probably feel how stressful that situation was for the people having to deal with it.  And how hard it was to keep customers trust and regain it. So although we don't all work  at companies that have such a big scale as good love over Amazon, if we create bugs,  they will affect the lives of our users. So it doesn't matter if we have millions or  thousands of hundreds of users, it will always be stressful. So that's why, like, basically  the most important, I mean, that takes me to the most important reason why I wanted  to give this talk and talk about my fuck ups. And it's because we learn more from failure  than from success. So when we fail, when we, like, when everything goes wrong or something  that we didn't expect to go wrong, goes wrong, it's where we learn the most. So I also want  to share a little bit about what I learned. So before we move on, and I tell you my story,  I want you to take a few seconds to maybe close your eyes if you want and think what's  the worst thing that could happen to you at work? Imagine what could be like the apocalypse  for you. So, okay, keep that in mind. And I'll tell you what my overnight failure story  was. So how did it happen? How did the overnight failure become a thing? So I used to work at  a company that had a, like, product that allowed people to carpool. It was a mobile app. And  so the mobile app allowed people to use their car to go to work every day. So for example,  let's say this is Mary. So Mary had a car. She used it to go to work and back every day.  And then the app I helped build allowed Mary to find Anna who had a similar, who didn't  have a car and also had a similar commute. So they had, like, a similar route and the  app allowed them to exchange money with one another. So when they, like, shared a car  to go to work, the app kept the record of the trips they took together and even allowed  them to be matched with someone else, like, for example, John, who also had a similar  commute. So they could ride share or share their car every day to go to work and the  app would keep track of every trip they took. So how the money exchange part worked is that,  like, once a week, there was a process that run that basically charged the passengers  and then got the money and paid the driver, paid whoever it was. So for example, in this  case, Anna. So going a little bit deeper, explain how this worked technically, how it  was implemented, so the process that run once a week looked kind of like this. So every  week, like, a job was triggered and the job went to the database and checked, like, every  trip that was, every trip that users took during the previous week and then for each  driver-passenger combination, it put, like, a job on a queue. We could call it like that.  So then it created a lot of jobs for each week and each one of these jobs contained,  like I said, mentioned, a passenger-driver combination and the total for the trips that  the passenger should pay the driver. So after that, what happened is that each job was processed  and the passenger was charged using a payment gateway and then the driver was paid and what's  there on the circle was basically managed by the payment gateway. It wasn't managed  by our system. The payment gateway took care of all of that. So I don't know if you're  still with me, so let's do a quick recap. So basically, we had an app that users used  to call Carpool every day that took care of the payment process each week. Passengers  were charged and drivers were paid for the trips. So now that you kind of understand  how it worked, I'm going to tell you exactly what happened. So it was, and given Sunday,  it was 6 a.m. where the process was triggered, when the process was triggered, and started  usual, the process runs started, like, yeah, the process runs started the marked start  of the day that I like to call Black Saturday. And it has nothing to do with, like, big discounts  and stuff, like shopping spree, a really fun day. It has to do with a really, really stressful  day. So I'll tell you what happened next. So then, let's say you have a user that went  really early to the store to buy, like, some excellent bread for her breakfast, for her  family, and then when she tried to pay, her card was declined. And she was like, well,  this is weird. This shouldn't happen. And this happened, like, around 6.25 a.m. So she  went online and investigated what was going on with her credit card, and she saw a lot  of charges that were made by the car pulling company I used to work for on that same morning.  So that's the second step of her, like, Black Saturday. So that was bad. But then things  started to look even worse because, like, more users were affected by this. And it was  not just a few users, it was a bunch of users. So the customer care team started to get,  like, a huge influx of complaints. It was bad. So let's say that at, like, 6.35, a lot  of users in, like, a small amount of time reported, like, a lot of bugs. So that's,  I mean, a lot of problems, a lot of charges. So that's where things definitely started  to look bad. And so it was 6.34. I was sleeping. I don't know about you, but I'm always sleeping  on Saturday at that time. So I was happily sleeping in my bed, and then the phone rang.  And it was a call from my boss. And this is roughly how the conversation went. My boss  told me, like, hey, sorry to wake you up this early, but there's, like, an issue in production.  There's a lot of customers complaining about it. And I was like, okay, sure, like, I'll  take a look right away. I was really stressed. I was trying to act cool. So I thought we had  ended the call, and I was like, fuck! Fuck! And then my boss said, like, hey, I'm still  in the line. It wasn't looking promising. So then, okay, this is how are they, like,  black-sided with looking. So it wasn't even 7 AM. It was already crappy. So I started  to look into what was going on, and I checked the payment gateway, and I definitely saw  a lot of duplicates charges. So the first thing I noticed was that this didn't make  any sense. So when I looked at our system, like our billing system, I saw that the queue  was still full of jobs. And so the first thing I thought I was doing was, like, to kill that  part of the process, so basically stop processing jobs. And that will at least still not create  more charges. But then I looked at the queue again, and the queue grew. This was bad. So  then I decided to basically stop the whole process so that no more jobs were put into  the queue. And then the first thing I thought about was, okay, let's refund all the charges  that we shouldn't have made. And this sounded like a really good idea at the moment, but  you'll see that later maybe it wasn't. So then another thing I noticed is that a lot  of the charges we had already made created a lot of transfers to the drivers, and those  were still also happening. So I also had to go and reverse all those transfers and stop  the ones that were remaining. So we could say that at, like, 7.28, the problem was kind  of contained at this point. So, yeah, the day wasn't still, was still not looking very  good. The queue was full. It was full of jobs, had thousands of jobs to be processed. And  when I started looking at them, I started noticing a pattern. So there were a lot of  duplicated jobs, as you might have expected. A lot of them contained the same user that  was a passenger, the same user that was a driver, and the same amount to be charged.  So yeah, there were a lot of duplicated ones. So what I noticed is that, like, something,  I mean, to me, it was clear that the problem was in two points of the system. The problem  was that here on this part of the system, it seems like we were putting every passenger  driver pair on the queue thousands of times for some reason. And also that for some reason,  when we processed the jobs, we didn't check if we had already charged that passenger-driver  combination. We still charged them, no matter what. So after debugging, finding what was  going on, finding what was the best to solve this, I wrote some tests that failed and wrote  the code to make them pass, deployed it, and it was all kind of fixed, or fixed. But that  took a long time. It was a long day. I felt really dumb, really frustrated. And when I  was about to deploy the code I wrote, I did the programmer's prayer that Aaron taught  to us on RailsConf this year. So please work, please work, please work. Fortunately, it  did. So at 10.55 p.m., I started doing the obvious thing, started looking for a new job.  Well, that was not true, but I was really frustrated. This is how I felt. Yeah, pretty much. Okay,  so this is what had happened. So there were thousands of users affected by the bug. All  users were charged a different amount, the worst case being one that was charged over  $5,000. And when we should have only charged a user 50, some users were charged up to 500  times. And the worst part was that we maxed out a lot of credit cards and also emptied  a lot of savings accounts. So as I had mentioned, I created a refund for every extra charge  with it. And the problem was that refunds take a long time. And people didn't have any  money on their banks account. So I'll tell you in a few minutes how we dealt with that.  I mean, something I was really glad about that is that we were still not on Ruby 3. Because  if so many jobs were processing such a little amount of time, I couldn't imagine what would  have happened if we were using that. The problem would have been much, much terrible. So as  I was mentioning, refunds take up to five business days. So there was unacceptable for  a lot of our users. So what we had to do is reach out to all of them and offer them an  expedited way to reimburse them, like an expedited option. Some of them were just sending them  checks or kind of not be a paypal. We did whatever we could and then figured out what  to do with the refund. But this was really stressful. I spent, like, I remember at least  the next week just grabbing information about how bad we have fucked up. Yeah, it was really  bad. So why did I want to tell you all of this? Like, why is this important? I want to go  back to the why. So embarrassing things will happen. It will eventually happen to all of  us I've mentioned before. And you might think that tests might save you from this, but they  won't. We had tests. That could review might save you from this, but it won't. We had  that as well. Southwards built by humans. And as Olivier mentioned on his talk, I think  on Wednesday, we're all humans. We make mistakes. So bad things will happen eventually. And  we need to make admitting mistakes easy for everyone. It's really important to create  a culture where admitting mistakes is fine. And you don't feel like you have to blame  others or to make excuses up for a mistake you made. And we need to be trust that we  won't be judged. Because, like, I mean, the safer people feel about admitting their mistakes,  the more they will learn about them and the more their colleagues will learn about them.  The more we as a community will learn about it. So when you're dealing with this sort  of situation, like my first recommendation is that you make sure before doing anything  that you understand. You really understand what happened. It's really easy to jump to  quick conclusions and do, like, whatever you think is the easiest fix or the most quick  fix for something. And sometimes we end up making the problem worse as it happened with  the refunds. It was a headache, basically, to deal with the people that we sent money  in, like, a different type of form. And then we had already, like, done a refund for. It  was crazy. It was a lot of, we had to go into, like, a lot of trouble and talk to the payment  gateway a lot. By the way, like, I don't know how the payment gateway allows us to do this.  Like, at least you have noticed that we were doing something weird. But anyway. My second  recommendation would be move really slow. So don't rush. The problem, I mean, there's,  you really already screwed up. The problems are ready. People already noticed the problem.  So it doesn't mean that the quicker you fix it, I mean, that if you fix it quick, like,  some people will notice. That's not the case. So just make sure that you take the time to  think what you're really doing, which really goes back to the thing I just mentioned, the  first one I just mentioned. Something that I really also learned and that I think a lot  of companies do a very good job doing this is document the problem, document what happened,  document what the root cause of it was, document how you fixed it, document what you will do  to prevent this from happening again. So also document the fix. And document, this is the  most important thing, I think, document the lesson learned. Because that's the only way  that you will really prevent stuff from happening again. And also, don't do this. Don't get  blamed. It's not about who wrote the code where the bug was. That's not important. The important  thing is that the process failed, the company failed, the team failed. It's not only the  follow the person who wrote the code. That's just part of it. So doing it, blaming is just  like not creating a good culture where people will feel safe and will feel like trust to  admit their mistakes. And really importantly, you are not your failures. Don't think that  because you failed, you did a big screw up. You're the worst developer ever because you're  not. Everyone has done it. Every developer you admire, they have screwed up at some point,  believe me. And I think chat follow can put it better than me. Like no one will care about  your, the bugs you create when you die. So keep in mind that it's so temporary. So as  bad as you might feel at the moment that you're dealing with a really bad situation, it will  pass. And you will look at it in the past and laugh about it, give talks about it. We  all, we're here, I mean, I really want to celebrate our failures because they also make  us what we do. So you can come and talk to me, tell me your failures. I've heard a lot  and it's a lot of fun. It's kind of therapeutic. And also you can tweet using the hashtag of  the conference. It will be great if we share our war stories, our scars. Because as a community,  we can help each other and like get us through this like really stressful moments in our  careers. We can make it together. And I'll actually let John and P.A. say it better. Oh,  too bad you can hear that. So anyway, I think that's all I wanted to say for today. I work  for this, oh wait, what happened? Okay, we're back. Okay, I work for this company called  Kupad. Go and check it out if you like cooking and sharing your recipes. We're hiring as  everyone else. So if you want to work at a company that works, sorry, where you can  build like an application that's used by millions of people around the world, come  and talk to me and thanks a lot.
The_overnight_failure_0_47.png The_overnight_failure_48_287.png The_overnight_failure_288_827.png The_overnight_failure_828_1259.png The_overnight_failure_1260_1571.png The_overnight_failure_1572_1667.png The_overnight_failure_1668_2027.png The_overnight_failure_2028_2735.png The_overnight_failure_2736_3683.png The_overnight_failure_3684_5231.png The_overnight_failure_5232_5543.png The_overnight_failure_5544_5927.png The_overnight_failure_5928_7379.png The_overnight_failure_7380_7559.png The_overnight_failure_7560_7883.png The_overnight_failure_7884_8051.png The_overnight_failure_8052_9695.png The_overnight_failure_9696_9767.png The_overnight_failure_9768_10379.png The_overnight_failure_10380_10883.png The_overnight_failure_10884_11111.png The_overnight_failure_11112_11243.png The_overnight_failure_11244_15335.png The_overnight_failure_15336_16055.png The_overnight_failure_16056_16583.png The_overnight_failure_16584_18323.png The_overnight_failure_18324_18623.png The_overnight_failure_18624_20051.png The_overnight_failure_20052_22043.png The_overnight_failure_22044_22163.png The_overnight_failure_22164_25955.png The_overnight_failure_25956_32627.png The_overnight_failure_32628_37187.png The_overnight_failure_37188_37727.png The_overnight_failure_37728_38483.png The_overnight_failure_38484_38639.png The_overnight_failure_38640_38673.png
Okay, so hi everyone. I hope you're having a good time here. I know this can be a hard time of the day. I'm the only team between Yon lunch and the spring two days of talks. Last night was fun. We had an amazing time at Ruby karaoke. So I'll try to make this entertaining for you. So this is actually my first time at RubyConf. It's also my first time in New Orleans. And I have really enjoyed my time here. But as Aaron, I think, just pointed out, so far the conference team seems to have been about death. That's a bit sad. I really don't think Ruby is dead. But just so we are all clear, let's check the internet because that's where you check when you want to find the sort of truth for something. So let's see it. Okay. Yeah, Ruby is not dead yet. Thanks for building the website, Jason. Yeah, we're safe. Okay, so now that we got that of the way, we can continue. So let me introduce myself. My name is Sebastian. I work at a company called Cookepad. I like when people treat me during my talks. So that's my true handle. I came all the way from Columbia. And if not because it's everything you know about Columbia, I think you should definitely go and visit. And have, like, know the real Columbia. And in fact, like, I have the perfect excuse for you to go there. We have a Ruby conference there. Next one is going to be September 2018. We'll be announcing the exact date really soon. So you can follow the conference on Twitter. It's going to be fun. So lately I've been, like, trying to get into stuff different from programming. So I've been taking some cooking lessons to improve my culinary skills. So when I was accepted to give the suck here, I decided that I wanted to prepare some jam alaya because it's something I've always wanted to try. So I went online, looked for a good recipe. Oh, that's it. Okay, it looks weird there. It looked nice, so I went ahead and bought all the ingredients, prepared everything, followed the steps, and there was a final result. As you can see, it was an epic fail. I definitely have a long way to go. But before we continue, I really want to ask everyone a favor. I'm really happy to be here, so I would really love to take a selfie with you. So if you could all, like, raise your arms, make some noise. I'll do this real quick, I promise. Woo! Yeah, thanks! You're great. Awesome. So now let's get a bit more serious. The name of the talk is the overnight failure. And this talk is based on a true story. Something happened to me on a previous show I had. So why do I want to talk about this? Why do I want to talk to tell everyone about a time I really fucked up? Well, there are a few reasons, and I'll share the most important with you. So first of all, we all have broken the internet at some point. We all have screwed up really bad. We all have created bugs that made introduction at some point. And if it hasn't happened to you yet, it will happen, believe me. It's just a matter of time. And also because we don't normally talk about this in public. Like, at conferences, like meetups, whatever, we see people talking about what they learned, how they built a cool thing, how they made a project be successful. Basically, we mostly hear about success stories. And I think people should really talk more about this in public because, I mean, it will make talking about screw ups easier for everyone. And we normally just talk about this with people that are really close to us. We do it in private, to our families, friends, or maybe our colleagues because they're on the same boat with us. And also because by not speaking about failures in public, I think we boost posture syndrome, which is something terrible that a lot of us have suffered. And although sometimes companies talk about this publicly, for example, like in January, good love, sorry, good love, was down for several hours or the mistake that take a WS and a big chunk of the internet with it down for also a few hours. And I can imagine how stressful the situation was. I even remember how, like, their website showed that everything was right because the red icon that they should have been shown was also down because of S3. So it's something that's easy to laugh about. It's fun when we look at other people's experience. But if you really think about it, you can probably feel how stressful that situation was for the people having to deal with it. And how hard it was to keep customers trust and regain it. So although we don't all work at companies that have such a big scale as good love over Amazon, if we create bugs, they will affect the lives of our users. So it doesn't matter if we have millions or thousands of hundreds of users, it will always be stressful. So that's why, like, basically the most important, I mean, that takes me to the most important reason why I wanted to give this talk and talk about my fuck ups. And it's because we learn more from failure than from success. So when we fail, when we, like, when everything goes wrong or something that we didn't expect to go wrong, goes wrong, it's where we learn the most. So I also want to share a little bit about what I learned. So before we move on, and I tell you my story, I want you to take a few seconds to maybe close your eyes if you want and think what's the worst thing that could happen to you at work? Imagine what could be like the apocalypse for you. So, okay, keep that in mind. And I'll tell you what my overnight failure story was. So how did it happen? How did the overnight failure become a thing? So I used to work at a company that had a, like, product that allowed people to carpool. It was a mobile app. And so the mobile app allowed people to use their car to go to work every day. So for example, let's say this is Mary. So Mary had a car. She used it to go to work and back every day. And then the app I helped build allowed Mary to find Anna who had a similar, who didn't have a car and also had a similar commute. So they had, like, a similar route and the app allowed them to exchange money with one another. So when they, like, shared a car to go to work, the app kept the record of the trips they took together and even allowed them to be matched with someone else, like, for example, John, who also had a similar commute. So they could ride share or share their car every day to go to work and the app would keep track of every trip they took. So how the money exchange part worked is that, like, once a week, there was a process that run that basically charged the passengers and then got the money and paid the driver, paid whoever it was. So for example, in this case, Anna. So going a little bit deeper, explain how this worked technically, how it was implemented, so the process that run once a week looked kind of like this. So every week, like, a job was triggered and the job went to the database and checked, like, every trip that was, every trip that users took during the previous week and then for each driver-passenger combination, it put, like, a job on a queue. We could call it like that. So then it created a lot of jobs for each week and each one of these jobs contained, like I said, mentioned, a passenger-driver combination and the total for the trips that the passenger should pay the driver. So after that, what happened is that each job was processed and the passenger was charged using a payment gateway and then the driver was paid and what's there on the circle was basically managed by the payment gateway. It wasn't managed by our system. The payment gateway took care of all of that. So I don't know if you're still with me, so let's do a quick recap. So basically, we had an app that users used to call Carpool every day that took care of the payment process each week. Passengers were charged and drivers were paid for the trips. So now that you kind of understand how it worked, I'm going to tell you exactly what happened. So it was, and given Sunday, it was 6 a.m. where the process was triggered, when the process was triggered, and started usual, the process runs started, like, yeah, the process runs started the marked start of the day that I like to call Black Saturday. And it has nothing to do with, like, big discounts and stuff, like shopping spree, a really fun day. It has to do with a really, really stressful day. So I'll tell you what happened next. So then, let's say you have a user that went really early to the store to buy, like, some excellent bread for her breakfast, for her family, and then when she tried to pay, her card was declined. And she was like, well, this is weird. This shouldn't happen. And this happened, like, around 6.25 a.m. So she went online and investigated what was going on with her credit card, and she saw a lot of charges that were made by the car pulling company I used to work for on that same morning. So that's the second step of her, like, Black Saturday. So that was bad. But then things started to look even worse because, like, more users were affected by this. And it was not just a few users, it was a bunch of users. So the customer care team started to get, like, a huge influx of complaints. It was bad. So let's say that at, like, 6.35, a lot of users in, like, a small amount of time reported, like, a lot of bugs. So that's, I mean, a lot of problems, a lot of charges. So that's where things definitely started to look bad. And so it was 6.34. I was sleeping. I don't know about you, but I'm always sleeping on Saturday at that time. So I was happily sleeping in my bed, and then the phone rang. And it was a call from my boss. And this is roughly how the conversation went. My boss told me, like, hey, sorry to wake you up this early, but there's, like, an issue in production. There's a lot of customers complaining about it. And I was like, okay, sure, like, I'll take a look right away. I was really stressed. I was trying to act cool. So I thought we had ended the call, and I was like, fuck! Fuck! And then my boss said, like, hey, I'm still in the line. It wasn't looking promising. So then, okay, this is how are they, like, black-sided with looking. So it wasn't even 7 AM. It was already crappy. So I started to look into what was going on, and I checked the payment gateway, and I definitely saw a lot of duplicates charges. So the first thing I noticed was that this didn't make any sense. So when I looked at our system, like our billing system, I saw that the queue was still full of jobs. And so the first thing I thought I was doing was, like, to kill that part of the process, so basically stop processing jobs. And that will at least still not create more charges. But then I looked at the queue again, and the queue grew. This was bad. So then I decided to basically stop the whole process so that no more jobs were put into the queue. And then the first thing I thought about was, okay, let's refund all the charges that we shouldn't have made. And this sounded like a really good idea at the moment, but you'll see that later maybe it wasn't. So then another thing I noticed is that a lot of the charges we had already made created a lot of transfers to the drivers, and those were still also happening. So I also had to go and reverse all those transfers and stop the ones that were remaining. So we could say that at, like, 7.28, the problem was kind of contained at this point. So, yeah, the day wasn't still, was still not looking very good. The queue was full. It was full of jobs, had thousands of jobs to be processed. And when I started looking at them, I started noticing a pattern. So there were a lot of duplicated jobs, as you might have expected. A lot of them contained the same user that was a passenger, the same user that was a driver, and the same amount to be charged. So yeah, there were a lot of duplicated ones. So what I noticed is that, like, something, I mean, to me, it was clear that the problem was in two points of the system. The problem was that here on this part of the system, it seems like we were putting every passenger driver pair on the queue thousands of times for some reason. And also that for some reason, when we processed the jobs, we didn't check if we had already charged that passenger-driver combination. We still charged them, no matter what. So after debugging, finding what was going on, finding what was the best to solve this, I wrote some tests that failed and wrote the code to make them pass, deployed it, and it was all kind of fixed, or fixed. But that took a long time. It was a long day. I felt really dumb, really frustrated. And when I was about to deploy the code I wrote, I did the programmer's prayer that Aaron taught to us on RailsConf this year. So please work, please work, please work. Fortunately, it did. So at 10.55 p.m., I started doing the obvious thing, started looking for a new job. Well, that was not true, but I was really frustrated. This is how I felt. Yeah, pretty much. Okay, so this is what had happened. So there were thousands of users affected by the bug. All users were charged a different amount, the worst case being one that was charged over $5,000. And when we should have only charged a user 50, some users were charged up to 500 times. And the worst part was that we maxed out a lot of credit cards and also emptied a lot of savings accounts. So as I had mentioned, I created a refund for every extra charge with it. And the problem was that refunds take a long time. And people didn't have any money on their banks account. So I'll tell you in a few minutes how we dealt with that. I mean, something I was really glad about that is that we were still not on Ruby 3. Because if so many jobs were processing such a little amount of time, I couldn't imagine what would have happened if we were using that. The problem would have been much, much terrible. So as I was mentioning, refunds take up to five business days. So there was unacceptable for a lot of our users. So what we had to do is reach out to all of them and offer them an expedited way to reimburse them, like an expedited option. Some of them were just sending them checks or kind of not be a paypal. We did whatever we could and then figured out what to do with the refund. But this was really stressful. I spent, like, I remember at least the next week just grabbing information about how bad we have fucked up. Yeah, it was really bad. So why did I want to tell you all of this? Like, why is this important? I want to go back to the why. So embarrassing things will happen. It will eventually happen to all of us I've mentioned before. And you might think that tests might save you from this, but they won't. We had tests. That could review might save you from this, but it won't. We had that as well. Southwards built by humans. And as Olivier mentioned on his talk, I think on Wednesday, we're all humans. We make mistakes. So bad things will happen eventually. And we need to make admitting mistakes easy for everyone. It's really important to create a culture where admitting mistakes is fine. And you don't feel like you have to blame others or to make excuses up for a mistake you made. And we need to be trust that we won't be judged. Because, like, I mean, the safer people feel about admitting their mistakes, the more they will learn about them and the more their colleagues will learn about them. The more we as a community will learn about it. So when you're dealing with this sort of situation, like my first recommendation is that you make sure before doing anything that you understand. You really understand what happened. It's really easy to jump to quick conclusions and do, like, whatever you think is the easiest fix or the most quick fix for something. And sometimes we end up making the problem worse as it happened with the refunds. It was a headache, basically, to deal with the people that we sent money in, like, a different type of form. And then we had already, like, done a refund for. It was crazy. It was a lot of, we had to go into, like, a lot of trouble and talk to the payment gateway a lot. By the way, like, I don't know how the payment gateway allows us to do this. Like, at least you have noticed that we were doing something weird. But anyway. My second recommendation would be move really slow. So don't rush. The problem, I mean, there's, you really already screwed up. The problems are ready. People already noticed the problem. So it doesn't mean that the quicker you fix it, I mean, that if you fix it quick, like, some people will notice. That's not the case. So just make sure that you take the time to think what you're really doing, which really goes back to the thing I just mentioned, the first one I just mentioned. Something that I really also learned and that I think a lot of companies do a very good job doing this is document the problem, document what happened, document what the root cause of it was, document how you fixed it, document what you will do to prevent this from happening again. So also document the fix. And document, this is the most important thing, I think, document the lesson learned. Because that's the only way that you will really prevent stuff from happening again. And also, don't do this. Don't get blamed. It's not about who wrote the code where the bug was. That's not important. The important thing is that the process failed, the company failed, the team failed. It's not only the follow the person who wrote the code. That's just part of it. So doing it, blaming is just like not creating a good culture where people will feel safe and will feel like trust to admit their mistakes. And really importantly, you are not your failures. Don't think that because you failed, you did a big screw up. You're the worst developer ever because you're not. Everyone has done it. Every developer you admire, they have screwed up at some point, believe me. And I think chat follow can put it better than me. Like no one will care about your, the bugs you create when you die. So keep in mind that it's so temporary. So as bad as you might feel at the moment that you're dealing with a really bad situation, it will pass. And you will look at it in the past and laugh about it, give talks about it. We all, we're here, I mean, I really want to celebrate our failures because they also make us what we do. So you can come and talk to me, tell me your failures. I've heard a lot and it's a lot of fun. It's kind of therapeutic. And also you can tweet using the hashtag of the conference. It will be great if we share our war stories, our scars. Because as a community, we can help each other and like get us through this like really stressful moments in our careers. We can make it together. And I'll actually let John and P.A. say it better. Oh, too bad you can hear that. So anyway, I think that's all I wanted to say for today. I work for this, oh wait, what happened? Okay, we're back. Okay, I work for this company called Kupad. Go and check it out if you like cooking and sharing your recipes. We're hiring as everyone else. So if you want to work at a company that works, sorry, where you can build like an application that's used by millions of people around the world, come and talk to me and thanks a lot.
serious. The name of the talk is the overnight because the red icon that they should have been wrong or something that we didn't expect to go worst thing that could happen to you at work? driver pair on the queue thousands of times for bug. All users were charged a different amount, the worst case being one that was charged over and then figured out what to do with the refund. will learn about them and the more their dealing with this sort of situation, like my document what you will do to prevent this from