You shipped it, you fix it
Cohen, Ron
This talk will explain and showcase how improving transparency and accountability in development teams can significantly improve the culture in the team and improve the quality of the code that gets released.
 Music Glad you can make it. This is the last talk of the day and I'll be sure to make it short so we can all get to the beers. You probably seen this graph when you came in. It's obviously not entirely true, not even remotely true actually. This is a good point of departure for our conversation. So by time it's called you ship it, you fix it, and it's going to be a whirlwind of advice or experience that we've gathered over the last couple of years. Hopefully you'll be able to avoid some of the pitfalls that we ran into. So how many of you guys work in an organization that looks like this? Well, operations is totally separate from development. See if you guys, yeah. So you've probably run into this problem where developers wrote some code, they give it to the apps people, it burns up. It's actually a pretty bad problem. The problem is sort of this, developers write the code, they give it to the apps people, if something breaks the apps people get paged and they have to figure out if they can fix the code or if they need to wake up a developer. And it's kind of, you know, the problem is accountability here. If you know that someone else is going to be paged when your code breaks, you're just going to write slightly worse code than you would if you were going to get paged if the thing breaks. This is human nature, I guess. So we've employed a policy of if you ship it, then you fix it. And a prerequisite for this to be useful is that if you build it, then you ship it. So you ship it, sorry, you build it, you ship it, you ship it, you fix it. So you're accountable for the whole process, the whole way from building code to actually making sure it runs in production. So is this DevOps? Maybe. Who knows at this point what DevOps is? Everybody is kind of not in agreement about what it is. So I Google it. And this is, there's a lot of different opinions on what DevOps is. These are some of the ones that kind of were mentioned the most, I guess. And they're pretty good. I like them. So empathy makes a lot of sense. Empathy across departments, trust and respect. And working towards common goals and breaking down silos in the organization. That all makes a lot of sense. So this could be what DevOps looks like in a modern organization. You've outsourced most of the hard cooperation stuff to Rackspace, Amazon, the local, et cetera. And then you have developers, you can't see it, but down there is developers and there's perhaps also an office person. And they're working together and they're doing development, but also web operations. So this is an example of what DevOps in modern organizations could look like. So let's say you already got this DevOps thing figured out. You bought some DevOps for the team, everything is good. What can you do now to actually improve the situation? You got DevOps mailed. One of the key points to be found is that you need to increase transparency. And that means you need to be able, developers need to be able to figure out what's going on. And one important way to do that, or good ways to do that is to make sure that the developers have the tools they need. So good tooling is really, really important. Making sure that developers have access to the best tools they can find and that all developers have access to this, to these tools. Here's an example. It's pretty small up there on the screen, but the idea is that an error happens. You log into a system, you can see a stack trace, you can see who actually wrote the code. You can see when you run into production. So this is a snapshot of Upbeat. And it's a way to increase transparency. Yeah. So the part is you can see exactly who wrote the code when you run into production. Another example of increasing transparency is this. This is a screenshot from Slacklin. And what happened here is that Benny, who's a great guy, he accidentally broke the build and Venya tells him to start doing pushups. So he has to do 80 pushups now because he broke the build and Benny does like it. So it's another way of increasing transparency and making everybody aware of what's currently going on. Here's a screenshot from New York, which is also a great tool. This shows performance for a verification and making sure that everybody can go in and see how their code is performing right now. It's extremely important to making sure the developers care about performance. So a big part of this is making the information accessible to developers. It's hard to get to this information. They will usually just don't go and get it. Crazy. Here's another example of a way to increase transparency. Make sure everybody knows when new stuff goes into production. People will be aware of what's going on in production to actually get a ping that something is out in production. Another screenshot from our Slacklin. So that was a little bit about how you can increase transparency and how that affects accountability in the organization. Now I want to talk a little bit about releases and sizes of releases and frequency of releases. So you can either do big releases. This isn't even a big release by some standards, but by standards this is a huge release. It took six weeks to prepare this release and it has 77 commits and a bunch of changed lines. There's only two people involved in this release, so that's a good, that's a pretty good metric. This is a small release. Six minutes from the commit was made to the production. Only one commit and only one person in this release. So the idea is that if you make small releases, it's much easier to figure out who broke stuff because fewer people are usually involved. And what happens is that if you have one developer per release and you find yourself in this scenario where there's a release that's suddenly causing errors, you can go in and see immediately who knows how to fix this. Besides that, there's a bunch of good reasons why you should be doing small releases. For example, developers love to put stuff in production. It's a great feeling when you finally put things into production. So you'll have happier developers if you do small releases. But there's, it's a huge topic and I encourage you to go on and investigate even more. So you ship it, you fixed it, you build it, you ship it. Whereas Werner likes to say, from Amazon, you build it, you run it, which is sort of a relation of that. And running it here could mean a lot of things. We usually try to make sure we have to run as few things as possible ourselves. So we like to buy services so other people will run it for us. But still, you know, you get the point. You're responsible for the code you write. Okay. Now, to something a little bit different. Do you know this person? I'm just a developer. Just let me know what I should build and I'll build it. And a variation of that is I just build whatever the specs said. Specs today, I need a, but is that really how we want to work as developers? Developers are smart people. We know how systems work. If we're really good at our job, we also know how users think. So here's an example. Again, this is Benny. He caught this quickly. He's a great guy. There's a nice, nice spec here. We're changing how timestamps look in our product. And it's spec that pretty rigorously what it should look like. Now, there's this thing that I've highlighted where it says if the time is more than a year ago, it should include the year as well. And there are two ways you can interpret this. If it's more than a year ago from the time it shouldn't give the year, or if it's the same year, then it should not include the year. You know what I mean? So it's either 365 days ago or just within the same year. So we got this really extensive spec, but he still went to ask the product guy, what exactly do you mean by this? And it turns out that the most obvious interpretation of this was not the one that the product guy meant. So it's a way to, what I'm getting at is that developers shouldn't just be these code monkeys who get specs, throw out them, and then start to, start to code, right? Developers are smart people. We usually have a good idea about how things work. So why should we just stop at dev and ops? It's great that we're trying to break down the silos between the development department and the ops department, but why not break down some more silos? So also break down the silos between the developers and product development or the product team and break down the silos between the developers and the customer support team. There's a lot of advantages in these, and you can probably imagine where you can go if the developers have more insights into what's going on in the organization in general, the kinds of protocols that are coming in, what are the thoughts from the product development team on how things are going to look in the future. I actually called this dev ops product, dev ops product. Yeah, that's it. So in summary, dev ops, new order, maybe, does a lot of good, a lot of good ideas. If you can increase transparency in your development team, you'll have better accountability, and in the end you'll have better code, and everybody will be able to sleep more, which is great. You can use tool to increase transparency and affect the culture in your team, and you should try to break down silos, not just between development and operations, but between many of the departments in the organization. That leads to better products and better solutions. That's it. Some of this was slightly maybe controversial. I'd love to discuss it with you guys, girls. I'll be right up there. So, yeah, come talk to me.
 Music  Glad you can make it. This is the last talk of the day and I'll be sure to make it short  so we can all get to the beers. You probably seen this graph when you came in.  It's obviously not entirely true, not even remotely true actually. This is a good  point of departure for our conversation. So by time it's called you ship it, you fix it,  and it's going to be a whirlwind of advice or experience that we've gathered over the  last couple of years. Hopefully you'll be able to avoid some of the pitfalls that we ran into.  So how many of you guys work in an organization that looks like this? Well,  operations is totally separate from development. See if you guys, yeah. So you've probably run  into this problem where developers wrote some code, they give it to the apps people, it burns up.  It's actually a pretty bad problem. The problem is sort of this, developers write the code,  they give it to the apps people, if something breaks the apps people get paged and they have to  figure out if they can fix the code or if they need to wake up a developer. And it's kind of,  you know, the problem is accountability here. If you know that someone else is going to be paged  when your code breaks, you're just going to write slightly worse code than you would if you were  going to get paged if the thing breaks. This is human nature, I guess. So we've employed a policy of  if you ship it, then you fix it. And a prerequisite for this to be useful is that if you build it,  then you ship it. So you ship it, sorry, you build it, you ship it, you ship it, you fix it.  So you're accountable for the whole process, the whole way from building code to actually  making sure it runs in production. So is this DevOps? Maybe. Who knows at this point what DevOps is?  Everybody is kind of not in agreement about what it is. So I Google it. And this is, there's a lot  of different opinions on what DevOps is. These are some of the ones that kind of  were mentioned the most, I guess. And they're pretty good. I like them. So empathy makes a lot  of sense. Empathy across departments, trust and respect. And working towards common goals  and breaking down silos in the organization. That all makes a lot of sense. So this could be  what DevOps looks like in a modern organization. You've outsourced most of the hard cooperation  stuff to Rackspace, Amazon, the local, et cetera. And then you have developers, you can't see it,  but down there is developers and there's perhaps also an office person. And they're working together  and they're doing development, but also web operations. So this is an example of what DevOps  in modern organizations could look like. So let's say you already got this DevOps thing figured  out. You bought some DevOps for the team, everything is good. What can you do now to actually improve  the situation? You got DevOps mailed. One of the key points to be found is that you need to increase  transparency. And that means you need to be able, developers need to be able to figure out what's  going on. And one important way to do that, or good ways to do that is to make sure that the  developers have the tools they need. So good tooling is really, really important. Making sure  that developers have access to the best tools they can find and that all developers have access to  this, to these tools. Here's an example. It's pretty small up there on the screen, but the idea is that  an error happens. You log into a system, you can see a stack trace, you can see who actually wrote  the code. You can see when you run into production. So this is a snapshot of Upbeat. And it's a way  to increase transparency. Yeah. So the part is you can see exactly who wrote the code when you  run into production. Another example of increasing transparency is this. This is a screenshot from  Slacklin. And what happened here is that Benny, who's a great guy, he accidentally broke the build  and Venya tells him to start doing pushups. So he has to do 80 pushups now because he broke the  build and Benny does like it. So it's another way of increasing transparency and making everybody  aware of what's currently going on. Here's a screenshot from New York, which is also a great  tool. This shows performance for a verification and making sure that everybody can go in and see  how their code is performing right now. It's extremely important to making sure the developers  care about performance. So a big part of this is making the information accessible to developers.  It's hard to get to this information. They will usually just don't go and get it.  Crazy. Here's another example of a way to increase transparency. Make sure everybody knows when  new stuff goes into production. People will be aware of what's going on in production to actually  get a ping that something is out in production. Another screenshot from our Slacklin. So that was  a little bit about how you can increase transparency and how that affects accountability  in the organization. Now I want to talk a little bit about releases and sizes of releases and frequency  of releases. So you can either do big releases. This isn't even a big release by some standards,  but by standards this is a huge release. It took six weeks to prepare this release and it has 77  commits and a bunch of changed lines. There's only two people involved in this release, so that's a  good, that's a pretty good metric. This is a small release. Six minutes from the commit was made  to the production. Only one commit and only one person in this release. So the idea is that if  you make small releases, it's much easier to figure out who broke stuff because fewer people are  usually involved. And what happens is that if you have one developer per release and you find  yourself in this scenario where there's a release that's suddenly causing errors, you can go in and  see immediately who knows how to fix this. Besides that, there's a bunch of good reasons why you  should be doing small releases. For example, developers love to put stuff in production. It's  a great feeling when you finally put things into production. So you'll have happier developers  if you do small releases. But there's, it's a huge topic and I encourage you to go on and  investigate even more. So you ship it, you fixed it, you build it, you ship it. Whereas Werner  likes to say, from Amazon, you build it, you run it, which is sort of a relation of that.  And running it here could mean a lot of things. We usually try to make sure we have to run as  few things as possible ourselves. So we like to buy services so other people will run it for us.  But still, you know, you get the point. You're responsible for the code you write.  Okay. Now, to something a little bit different. Do you know this person?  I'm just a developer. Just let me know what I should build and I'll build it.  And a variation of that is I just build whatever the specs said. Specs today, I need a,  but is that really how we want to work as developers? Developers are smart people. We  know how systems work. If we're really good at our job, we also know how users think. So  here's an example. Again, this is Benny. He caught this quickly. He's a great guy.  There's a nice, nice spec here. We're changing how timestamps look in our product. And it's  spec that pretty rigorously what it should look like. Now, there's this thing that I've highlighted  where it says if the time is more than a year ago, it should include the year as well.  And there are two ways you can interpret this. If it's more than a year ago from the time it  shouldn't give the year, or if it's the same year, then it should not include the year. You know what  I mean? So it's either 365 days ago or just within the same year. So we got this really  extensive spec, but he still went to ask the product guy, what exactly do you mean by this?  And it turns out that the most obvious interpretation of this was not the one that the  product guy meant. So it's a way to, what I'm getting at is that developers shouldn't just  be these code monkeys who get specs, throw out them, and then start to, start to code, right?  Developers are smart people. We usually have a good idea about how things work.  So why should we just stop at dev and ops? It's great that we're trying to break down the silos  between the development department and the ops department, but why not break down some more silos?  So also break down the silos between the developers and product development or the product team  and break down the silos between the developers and the customer support team.  There's a lot of advantages in these, and you can probably imagine where you can go if the  developers have more insights into what's going on in the organization in general,  the kinds of protocols that are coming in, what are the thoughts from the product development team  on how things are going to look in the future. I actually called this  dev ops product, dev ops product. Yeah, that's it. So in summary, dev ops, new order, maybe,  does a lot of good, a lot of good ideas. If you can increase transparency in your development team,  you'll have better accountability, and in the end you'll have better code,  and everybody will be able to sleep more, which is great. You can use tool to increase transparency  and affect the culture in your team, and you should try to break down silos, not just between  development and operations, but between many of the departments in the organization.  That leads to better products and better solutions. That's it. Some of this was slightly  maybe controversial. I'd love to discuss it with you guys, girls. I'll be right up there. So, yeah,  come talk to me.
DjangoCon_2014-_You_shipped_it__you_fix_it__p_Iq2AnLoA_0_71.png DjangoCon_2014-_You_shipped_it__you_fix_it__p_Iq2AnLoA_72_143.png DjangoCon_2014-_You_shipped_it__you_fix_it__p_Iq2AnLoA_144_311.png DjangoCon_2014-_You_shipped_it__you_fix_it__p_Iq2AnLoA_312_419.png DjangoCon_2014-_You_shipped_it__you_fix_it__p_Iq2AnLoA_420_503.png DjangoCon_2014-_You_shipped_it__you_fix_it__p_Iq2AnLoA_504_4775.png DjangoCon_2014-_You_shipped_it__you_fix_it__p_Iq2AnLoA_4776_5531.png DjangoCon_2014-_You_shipped_it__you_fix_it__p_Iq2AnLoA_5532_6599.png DjangoCon_2014-_You_shipped_it__you_fix_it__p_Iq2AnLoA_6600_7391.png DjangoCon_2014-_You_shipped_it__you_fix_it__p_Iq2AnLoA_7392_11651.png DjangoCon_2014-_You_shipped_it__you_fix_it__p_Iq2AnLoA_11652_13427.png DjangoCon_2014-_You_shipped_it__you_fix_it__p_Iq2AnLoA_13428_15575.png DjangoCon_2014-_You_shipped_it__you_fix_it__p_Iq2AnLoA_15576_18323.png DjangoCon_2014-_You_shipped_it__you_fix_it__p_Iq2AnLoA_18324_18467.png DjangoCon_2014-_You_shipped_it__you_fix_it__p_Iq2AnLoA_18468_18697.png
Music Glad you can make it. This is the last talk of the day and I'll be sure to make it short so we can all get to the beers. You probably seen this graph when you came in. It's obviously not entirely true, not even remotely true actually. This is a good point of departure for our conversation. So by time it's called you ship it, you fix it, and it's going to be a whirlwind of advice or experience that we've gathered over the last couple of years. Hopefully you'll be able to avoid some of the pitfalls that we ran into. So how many of you guys work in an organization that looks like this? Well, operations is totally separate from development. See if you guys, yeah. So you've probably run into this problem where developers wrote some code, they give it to the apps people, it burns up. It's actually a pretty bad problem. The problem is sort of this, developers write the code, they give it to the apps people, if something breaks the apps people get paged and they have to figure out if they can fix the code or if they need to wake up a developer. And it's kind of, you know, the problem is accountability here. If you know that someone else is going to be paged when your code breaks, you're just going to write slightly worse code than you would if you were going to get paged if the thing breaks. This is human nature, I guess. So we've employed a policy of if you ship it, then you fix it. And a prerequisite for this to be useful is that if you build it, then you ship it. So you ship it, sorry, you build it, you ship it, you ship it, you fix it. So you're accountable for the whole process, the whole way from building code to actually making sure it runs in production. So is this DevOps? Maybe. Who knows at this point what DevOps is? Everybody is kind of not in agreement about what it is. So I Google it. And this is, there's a lot of different opinions on what DevOps is. These are some of the ones that kind of were mentioned the most, I guess. And they're pretty good. I like them. So empathy makes a lot of sense. Empathy across departments, trust and respect. And working towards common goals and breaking down silos in the organization. That all makes a lot of sense. So this could be what DevOps looks like in a modern organization. You've outsourced most of the hard cooperation stuff to Rackspace, Amazon, the local, et cetera. And then you have developers, you can't see it, but down there is developers and there's perhaps also an office person. And they're working together and they're doing development, but also web operations. So this is an example of what DevOps in modern organizations could look like. So let's say you already got this DevOps thing figured out. You bought some DevOps for the team, everything is good. What can you do now to actually improve the situation? You got DevOps mailed. One of the key points to be found is that you need to increase transparency. And that means you need to be able, developers need to be able to figure out what's going on. And one important way to do that, or good ways to do that is to make sure that the developers have the tools they need. So good tooling is really, really important. Making sure that developers have access to the best tools they can find and that all developers have access to this, to these tools. Here's an example. It's pretty small up there on the screen, but the idea is that an error happens. You log into a system, you can see a stack trace, you can see who actually wrote the code. You can see when you run into production. So this is a snapshot of Upbeat. And it's a way to increase transparency. Yeah. So the part is you can see exactly who wrote the code when you run into production. Another example of increasing transparency is this. This is a screenshot from Slacklin. And what happened here is that Benny, who's a great guy, he accidentally broke the build and Venya tells him to start doing pushups. So he has to do 80 pushups now because he broke the build and Benny does like it. So it's another way of increasing transparency and making everybody aware of what's currently going on. Here's a screenshot from New York, which is also a great tool. This shows performance for a verification and making sure that everybody can go in and see how their code is performing right now. It's extremely important to making sure the developers care about performance. So a big part of this is making the information accessible to developers. It's hard to get to this information. They will usually just don't go and get it. Crazy. Here's another example of a way to increase transparency. Make sure everybody knows when new stuff goes into production. People will be aware of what's going on in production to actually get a ping that something is out in production. Another screenshot from our Slacklin. So that was a little bit about how you can increase transparency and how that affects accountability in the organization. Now I want to talk a little bit about releases and sizes of releases and frequency of releases. So you can either do big releases. This isn't even a big release by some standards, but by standards this is a huge release. It took six weeks to prepare this release and it has 77 commits and a bunch of changed lines. There's only two people involved in this release, so that's a good, that's a pretty good metric. This is a small release. Six minutes from the commit was made to the production. Only one commit and only one person in this release. So the idea is that if you make small releases, it's much easier to figure out who broke stuff because fewer people are usually involved. And what happens is that if you have one developer per release and you find yourself in this scenario where there's a release that's suddenly causing errors, you can go in and see immediately who knows how to fix this. Besides that, there's a bunch of good reasons why you should be doing small releases. For example, developers love to put stuff in production. It's a great feeling when you finally put things into production. So you'll have happier developers if you do small releases. But there's, it's a huge topic and I encourage you to go on and investigate even more. So you ship it, you fixed it, you build it, you ship it. Whereas Werner likes to say, from Amazon, you build it, you run it, which is sort of a relation of that. And running it here could mean a lot of things. We usually try to make sure we have to run as few things as possible ourselves. So we like to buy services so other people will run it for us. But still, you know, you get the point. You're responsible for the code you write. Okay. Now, to something a little bit different. Do you know this person? I'm just a developer. Just let me know what I should build and I'll build it. And a variation of that is I just build whatever the specs said. Specs today, I need a, but is that really how we want to work as developers? Developers are smart people. We know how systems work. If we're really good at our job, we also know how users think. So here's an example. Again, this is Benny. He caught this quickly. He's a great guy. There's a nice, nice spec here. We're changing how timestamps look in our product. And it's spec that pretty rigorously what it should look like. Now, there's this thing that I've highlighted where it says if the time is more than a year ago, it should include the year as well. And there are two ways you can interpret this. If it's more than a year ago from the time it shouldn't give the year, or if it's the same year, then it should not include the year. You know what I mean? So it's either 365 days ago or just within the same year. So we got this really extensive spec, but he still went to ask the product guy, what exactly do you mean by this? And it turns out that the most obvious interpretation of this was not the one that the product guy meant. So it's a way to, what I'm getting at is that developers shouldn't just be these code monkeys who get specs, throw out them, and then start to, start to code, right? Developers are smart people. We usually have a good idea about how things work. So why should we just stop at dev and ops? It's great that we're trying to break down the silos between the development department and the ops department, but why not break down some more silos? So also break down the silos between the developers and product development or the product team and break down the silos between the developers and the customer support team. There's a lot of advantages in these, and you can probably imagine where you can go if the developers have more insights into what's going on in the organization in general, the kinds of protocols that are coming in, what are the thoughts from the product development team on how things are going to look in the future. I actually called this dev ops product, dev ops product. Yeah, that's it. So in summary, dev ops, new order, maybe, does a lot of good, a lot of good ideas. If you can increase transparency in your development team, you'll have better accountability, and in the end you'll have better code, and everybody will be able to sleep more, which is great. You can use tool to increase transparency and affect the culture in your team, and you should try to break down silos, not just between development and operations, but between many of the departments in the organization. That leads to better products and better solutions. That's it. Some of this was slightly maybe controversial. I'd love to discuss it with you guys, girls. I'll be right up there. So, yeah, come talk to me.
of the hard cooperation stuff to Rackspace, good. What can you do now to actually improve the transparency and how that affects accountability product development or the product team and break increase transparency and affect the culture in