How to turn software into competitive business advantage
Adzic, Gojko
As an industry, our capability for delivering software increased significantly over the last ten years. We now have much better processes, tools and techniques, and we can produce a lot more software a lot easier and faster. But companies rarely get the benefits from this new capability, because the ecosystem around software teams hasn't improved at the same speed. Gojko will talk about ideas how to unlock all that potential and turn the fact that your teams can deliver frequently into an engine that will help you outrun your competition. This is a talk for tech and business people.
 Are we on? Hello? Yeah, good. Fantastic. So, okay, the idea for this talk is basically to try and pack in one hour two ideas that if you think about those two ideas over the next five years, I kind of guarantee that you will be able to become a lot more important in your company. And although they're going to be very, very difficult to implement, if you persevere and do them over the next five years, I will guarantee that you will turn IT from a cost center to a value center in your company. So, kind of, that's not bad for a one hour of your time. And given that kind of, I've messed up my air travel planning, I'll have to run away about 10 minutes early, so it's not going to be even a whole hour. And from that perspective, we're going to look at a couple of things that I think are going to happen in the near future to most people here. And future, as always, is kind of elusive and uncertain. And although, you know, we don't exactly know what's going to happen in the future, we can pretty much talk about some things with certainty. For example, nobody knows when the world is exactly going to end, but pretty much anybody who's done any development over the last five years knows that the apocalypse is going to be caused by copy paste from Stack Overflow. So, some of those things are kind of more or less certain. And I think a couple of patterns are emerging now that I'm seeing with lots of different people that are going to be really, really interesting for the industry over the next five years. And one of them is what's happening with continuous delivery. So Ian Cooper did a really good talk yesterday. He talked about microservices and continuous delivery. And he said that kind of as stuff gets faster and faster, people's appetite for risk increases, because continuous delivery starts decreasing the risk of deployments and the risk of releases. And then people's appetite increases quite a lot. And I think that kind of changes dynamics in a couple of really interesting ways that software people don't necessarily see. So that's the first thing I want to talk about, kind of how do we start spotting these wider patterns? So kind of the key message for today is pretty much that the continuous delivery has serious side effects that software people don't often consider. And by looking at these side effects, by trying to kind of use them for our own purpose, we can open up new business opportunities, we can open up new markets, we can get a lot more value out of continuous delivery. At the same time, if we decide to ignore these things, and if we approach continuous delivery from a purely technical perspective, we can mess up really, really, really badly. And I can tell you about this from my own kind of personal experience I've kind of messed up pretty badly. The nice thing about coming to speak at the same conference over several years is I can always come and talk about my new mistakes and people can laugh about that. And I can feel like an alcoholic synonymous where every year I make new mistakes and every year there's something new to talk about. So kind of the stuff I want to talk about is mistakes I've made last year effectively that are in this area. So kind of lots of people, especially from our industry, when they look at continuous delivery, they look at kind of faster deployments, they look at the trend in going faster and faster and faster, they start thinking about, well, how can possibly this be bad? Because, you know, fast is good, isn't it? And if you look outside of software, there's a ton of examples where fast is not necessarily good. There's a ton of examples where fast stuff is really, really bad for you. And in a lot of these cases, kind of we tend to consider fast as good, but that's not necessarily true. For example, I worked with a TV station a couple of years ago where their developers started doing continuous delivery and they were moving from a cycle where they were no longer releasing every few months, they were releasing every few weeks and then every few days and then a couple of times a day. And what the developers thought they were giving the business is a shiny new toy that's really fast, really good, shiny new toy that's really fast, really good, that's kind of fantastic. Unfortunately, what the business users were seeing is this shiny new toy, but it's a completely different scenario where they were trapped inside something that was kind of changing too frequently and they really didn't want to use that. The problem here was that the technical platform was designed so that every time there's an update, everybody who's currently logged onto the system had to log off and re-log in again. And when they had an update every couple of months, that was no problem at all. When they were updating several times a day, the content managers were pissed off all the time. So, kind of this is something where two completely different opinions about continuous delivery emerged in a company where the business people hated it, technical people loved it and it's completely strange to kind of have something like that. So, one of the key reasons for this is a loss of human control. Fast is good when there's a human element of control over that. When we lose human control over that, when business users that are kind of content managers in this case have no say over what gets updated and when, because stuff continuously gets pushed to them, we've lost control. And loss of control is something that kind of sci-fi writers have been warning us about for decades now. You have lots of stuff where, you know, robots emerge and take over humanity and kill everybody. And if you want to go for a really bad scenario, then you have robots that address to look like humans. If you want to go for a really, really, really, really, really bad scenario, then you start thinking about robots that address like humans, but they look harmless. So, that's kind of one of the biggest problems we have now is I think continuous delivery is reaching this point where we really need to start thinking about can we control it, or are we losing control over that. And again, going back to sci-fi, Issa Kasimov solved this problem in 1948, and he proposed the three rules of kind of robotics to make sure that evil machines don't take over the world and rule the world. So, I have three copies of my book to give away to people who can remember at least one of these rules. Don't come a human, okay? It was the same, okay? So, no. Well, okay, but that's kind of don't come another human. So, that's so you get one. Do not bring harm to yourself. Well, kind of, I'll give you that. I'll give you that. That's kind of the third one where they have to continue their existence or protect their existence unless it's conflicting with rule number one. Okay, so, kind of the three rules that Issa Kasimov declared, the first of all, don't harm a human. The second one is obey everything humans tell you, unless it's conflicting rule number one, and the third one is kind of try and protect yourself unless kind of it's conflicting rule number one, rule number two. So, these three things are very, very important. We'll come back to that because I think that's part of the secret how we can start unlocking the real power of continuous delivery. Now, kind of in terms of what's been happening in the industry, especially over the last five or six years, I think we're getting to a point where the technical part of continuous delivery is kind of solved. But that's opening up some really, really interesting new possibilities. And I think this is very close to what happened when factories were first electrified. Before electric motors, what most factories used to have is they used to have a boiler room in the basement, and they were built to be very tall because steam goes up. That's how you harness steam. So, they would put a pipe there and then they would put lots of stuff on each floor to be powered by the same pipe. Now, when the first electric motors came in, people got something that was safer than steam, they got something that was more powerful, that was cheaper, but they used the same paradigms for that. They would put the electric motor in the basement, and they would power the whole thing. And it took 10 years for the industry to realize that the constraints of the old world, the steam going only up, have been solved and they no longer exist. And then they could kind of come up with lots and lots of interesting solutions for other problems. For example, when a factory is very tall, you have to solve a problem of transporting things up and down. And because that's no longer a constraint, people started building very, very flat factories where it's very easy to transport stuff because electricity goes wherever the cables go. And I think that's kind of the situation we are in now with continuous delivery, where we can start dismantling stuff we built around the constraints of the old world because they no longer exist. For example, in January last year, the American National Transport Safety Organization that regulates their kind of transport companies issued two product recalls, both related to potential fire hazards. One was for General Motors cars, and another one was for Tesla cars. And kind of, for the Tesla car, they issued a recall that was going to affect about 20,000 vehicles, and they were complaining about how a plug in the charger is a potential fire hazard. Now, General Motors had to bring all their cars back to a repair shop to kind of fix them, to send them back. Tesla said, no problem, we're going to do a software update, and it's going to happen overnight. So pretty much when people woke up next morning, this problem was solved. Now, this is an example of what I mean by we can start dismantling the constraints of the old world, because when you have a car that can self update over the internet to resolve a hardware problem, and that doesn't cost anybody almost anything. People don't have to drive their cars into a repair shop. Tesla doesn't have to employ people to fix it. This is now a huge, huge, huge competitive advantage where kind of General Motors has to fix cars, these guys issue a software patch. And this is the flat factory thing, this is happening now. And at the same time, continuous delivery starts creating some very, very unusual feedbacks on other parts of the business as well. For example, about two years ago, I started working on a collaboration tool with some friends, and we started doing stuff the way we're used to doing it. So we started with continuous delivery from day one, we were packaging stuff, releasing them very quickly. And this is our user growth graph over the last two years. Watching spot is somewhere towards the beginning, there's a massive spike. Then there's another kind of spike in another spike over there, very, very close to the beginning. And then for the next two years, no spikes at all. What happened there is we appeared on Hacker News on the front page, then what happened there is we appeared on Reddit on the front page. And then what happened there is the official version one was reviewed by PC World. What happened for the other one and a half years is we tried as much as we could to get anybody to write about us and failed. Although we did a ton of really, really interesting things, what we learned is that when you do continuous delivery, you take the drama out of the process. There's nothing newsworthy to write about. There's kind of nothing big to announce. There's no marketing opportunities anymore and kind of our marketing strategy that was to get blogs and newspapers to write about us was completely destroyed by the fact that we took the drama out of delivery. So this is a really interesting aspect that I never expected because previously when we did stuff like that, you'd release every six months or every year, you announced version 3.0, people write about that because journalists like to kind of write about something that's newsworthy. Now when you say, oh, we've updated this feature, we've updated that feature, there's nothing, nothing to come out of that. So kind of, although we perceived continuous delivery as a purely technical thing, it had a massive side effect on our marketing. Now that's not necessarily horribly bad. It's just that it prevented us from doing one marketing thing. We had to learn about that the hard way. And we can say, oh, I'm stupid because I make mistakes every year and I'm kind of coming here to talk to you about this. But other companies do this as well. And they get very, very surprised by the marketing side effects of continuous delivery because that's not a purely technical thing. For example, I'm going back to Tesla in October last year. Tesla announced a fantastic new feature that they're starting to put in their cars. And kind of Tesla with the whole continuous delivery model is trying to challenge this perception of models for cars. Ford, General Motors, Mercedes, they have models, Tesla kind of has a model S and just kind of ships updates. So they start announcing that, you know, from tomorrow morning, all model S cars that come out will have a front facing radar, 12 sonar sensors, several cameras, and they'll have an autopilot that's going to automatically change lanes. They'll have a digital break that's going to spot when there's a dangerous situation and kind of prevent stuff like that. So this is something that's amazing. It's kind of a fantastic safety feature. It's a fantastic feature anyway. And everybody was very excited about that. Newspapers wrote about how shit scared about this because when you think about Tesla doing automatic navigation, radar stuff, sensor stuff, you kind of know what the end game is going to be. So where kind of, I wasn't the only one that kind of felt negative about this. There was a guy called Richard Wolpert in the States that woke up one day, read the news about this, and he just felt that the world was unfair. Everything went wrong that day because he just bought a model S car that didn't have all these nice shiny features. And here's Tesla announcing that from tomorrow everybody else is going to get the radar, a sonar, and I don't know why they need a sonar. I guess they're preparing a car to go underwater. But kind of, you know, 12 sonars, two radars, cameras, automatic navigation, things like that. Richard felt that the world was just unfair. So what do people do when they feel the world is unfair? They start bitching about it online. And he started a petition to kind of force Tesla to get autopilot radars and sonars retrofitted into all the old models. And 981, 948 people signed up for this petition to kind of start forcing Tesla to retrofit stuff because people felt entitled. As continuous delivery teaches people that we can now ship stuff easily, update stuff, users start feeling more entitled. Now, I've never heard of anybody going to Mercedes or to Ford and saying, Oh, you have a new model. I bought the car five years ago. I want everything that the new car has for free. But Tesla taught their users to expect more. And, you know, people all over the world signed up for this. Where, you know, even in Norway, there was a guy in Norway called Jochen, who said, Hey, because it's been said, I want this stuff, I'm entitled to this stuff. And this is because kind of when we as we start doing continuous delivery, our stakeholders, our users start expecting stuff that might be completely unrealistic. And, you know, installing radars and sonars and stuff like that, you know, that's not a software update thing. That's kind of Star Trek replicator stuff. That's not something you can do very, very easily. But again, these people felt entitled. So regardless of whether that's possible or not possible, disappointing users is never good. Making users angry enough to start an online petition against you is never good. Tesla is still in an early adopter stage. And at that point, they were selling something like one and a half thousand cars a month. So if you have 1000 people pissed off enough to sign an online petition, and you're selling one and a half thousand cars a month, that's not good. So kind of this is where I think we need to start thinking about this slightly differently. And especially when kind of users are concerned, continuous delivery is typically something that technical people approach as a way to reduce technical risk. And users typically get at the wrong end of the stick. So if you're doing continuous delivery, please, please, please, please, consider your poor users. For example, I tend to travel a lot for conferences. And I was in Australia last year. Australia, unlike kind of normal countries, has stupidly expensive internet. I'm talking about, say, $50 for three gigabytes. And they sell it by gigabyte, they don't sell it by time based access and things like that. So as we were there, the conference organizers gave us these small kind of my five devices to use so that we don't pay the hotels. And I kind of like to fiddle with my slides until the very last moment I was changing the slides as people were coming into this room now. So about two hours before my talk in Australia, I finally sat down to do my slides. And normally that's not a problem at all. You go to Google Images, you find Patrick Stewart, you put it on, and that's perfect. That day, the world seemed unfair. None of the browsers seem to work. Everything seemed to crash all the time. My ping was 10 seconds to anything. And I couldn't figure out what was going on. After about half an hour of panicking, restarting the computer, trying to figure out whether Apple's network demon is a fault or whatever is going on, I've realized that my iPad decided on its own to update iOS and kind of took out the complete pipe I had. And not only that, but kind of spent all the bandwidth I had left. So I had to pay $50 for two more gigabytes. Then I had to find slides in a panic mode and kind of do with a half-finished presentation. So fair enough, that's my fault. It's my fault for not disabling automatic updates. But at the same time, Apple had an important update. And the update is never, ever, ever going to be as important as the work I have to do now. When people have an important security update, that's important for the company doing the update. That's basically saying, we cocked up, we're embarrassed, and now we want to update your system. If I'm doing something at that moment, that's not important at all. That's very low on the scale of importance. Likewise, for the content managers at TV Station, the technical update that was done for some other users somewhere half way around the world was completely unimportant to them. What was important is they were half way through writing a story, and now they have to log out and re-log in again. And I think we need to start considering that if we're not going to piece of users, if you really, really, really, really, really, really, really want a piece of users, then what you should do is deploy UX changes partially. And do continuous delivery on the UX. For example, PayPal did this to me about six months ago. They were starting to deploy the new business dashboard. Whenever you hear news somewhere, run away, run to the hills. So I woke up one day and there were a few thousand dollars missing from my company PayPal account. Now, that's never good. And because several people have access to the account, I didn't know whether somebody took the money out or whether they were going to go to where the kind of we got defrauded or something like that. So in panic mode, I started downloading transactions, figuring out what was going on, adding things up, but then they just didn't add up. If I added up the transactions, it looked fine. If I looked at the balance in the business dashboard, several thousand dollars were missing. So kind of I noticed a link on top. This is switched to classic and I put my testers hat down and I thought somebody must have caught something up. So I switched to classic and there was the money. So what effectively they did is we had multiple currencies in our accounts. And because the primary account was British pounds, they kind of deployed the UX changes gradually and they thought, well, we're just going to do it first for people with one account. And they were only showing the British balance. Now, I could put my testers hat on and I could notice the switch to classic thing, but I can bet you lots and lots of people that they had a really bad day and the call center supported people had a really bad day as well. So kind of that's why we're, you know, one of the biggest problems when we talk about continuous delivery, although continuous delivery is incredibly empowering. It opens up new business opportunities. It can kind of help you be Tesla, not general motors and stuff like that. When you go and talk to people about doing big stuff, they're really, really scared. An insurance company I worked with, when they started moving to continuous delivery, the business users thought, this is really brilliant. This is great. We can do small stuff easily, but our technology lost the capability of doing any big changes anymore because they just fight anything that's big. When you go and talk to UX people and you say, okay, we're going to start updating one page and we're going to update another, they start running for the hills because if we start looking at this from a development perspective, we think what we're doing is Amazon Prime. We think we're shipping stuff well. We think it's all fine. It's all brilliant. But when you look at it from a user perspective, from a UX perspective, what those people feel like is they're being force fed. And there's kind of a massive difference between continuous delivery this way and continuous delivery that way. So one of the most important things we need to start thinking about, especially kind of doing big, big changes and not losing the capability of doing big stuff is not doing this stuff over here. So kind of, I've almost always, when we start talking about continuous delivery to business people in terms of reducing technical risk, they love it. Then you go and talk to kind of how you're going to do a big technology refresh or how you're going to risk in the website, how you're going to rebrand something when you need, and people start fighting you with everything they can. UX designers kind of complain that you can't go and confuse users because if you start rebranding one part of the website, the other part is going to be completely confused. And that's what happened to us. I mean, I can't complain about that. And especially when you start thinking about slicing things, it's almost impossible to slice continuous delivery of UX in any sensible way. People will always be confused and it's always going to be stupid. So we can either not do continuous delivery or we can do it in a way that confuses people. And that's where kind of, this is really, really a big challenge. So one of the most important things to unlock the potential of this over the next five years is going to look at how do we not lose human control over this? How do we not go for false feeding? How do we go for Amazon Prime? And how do we, if we have to do this, because we'll have to do it. But how do we at least keep human control over that? So we do it in a sensible way because there are sensible ways of doing this. So kind of going back to the three rules of robotics, I think kind of what I'd like to rewrite this, so we're not going to talk about robots anymore, we'll talk about continuous delivery. And the first rule that talks about may not injure a human. Fair enough, if people are doing kind of medical devices, I hope you already have this in your testing. But when we talk about continuous delivery, particularly when we talk about UX, I think the biggest problem here is confusion. So the first rule of continuous delivery that we really have to put in place is this can't confuse users. Now, the second rule talks about obeying orders for humans, unless kind of order one, and I'm going to kind of skip the unless order, unless rule one. I think kind of this thing may must obey users when we think about what people actually want in terms of continuous delivery. Most often, they don't want it. If I'm in the middle of something, I do not want the ground to move. I want the stuff to stay the same. And I want this stuff, if it has to happen, to happen without me noticing at all. So the second rule of continuous delivery, if we're going to do it right, I'm going to replace must obey orders from must not interrupt users' work or sessions, because that's the really, really important thing. If that happens, I don't care about. Now, the third one is interesting kind of must protect its own existence. And where continuous delivery fails to protect its own existence, if it starts fighting the business. If there's a business reason why we want to risk in the website, or there's a business reason why we want to do technology refresh, we want to move from Oracle to in-memory data grids, we want to do a huge change that is going to involve rewriting everything to work on a mobile and things like that. Continuous delivery cannot fight the business there. If we start saying, oh, you know, that's possible, but only if we start moving people and for a while everything's going to be messed up, that's not going to work. And one of the most important things there is, I think, continuous delivery has a huge potential to disrupt marketing. That's what we felt on our own scale. That's what Tesla felt. So I think kind of continuous delivery rule number three in terms of protecting its own existence must be not to disrupt or prevent marketing initiatives. And if we can do this kind of stuff, then we can put human control around all this robotic mechanism and we can actually benefit quite a lot from the whole thing. So when we started looking at these three things for the collaboration tool I mentioned, we realized very, very quickly that we had a fundamental misunderstanding of two really important concepts. In our mind, deployment and release were the same thing. And for most software people, and it's likely that most of in the audience are now kind of scratching your heads, deployment and release are the same thing. But kind of we realized that's completely wrong. And that's one of those constraints of the old world. That's a tall factory. And if we can move away from that because those constraints no longer apply, then we can get really, really, really, really interesting benefits out of the whole thing. So the first thing that I'd really like you to kind of remember from this is that deployment isn't and shouldn't be the same thing as a release. Deployment is a technical event of pushing software to production. And I'm really thinking production here. I don't care about deployment to staging. I don't care about deployment to development environments. That's all nice and fine. But deployment is deployment to production. That's a technical event of moving code to production. Now, most of you, I assume, have been kids at some point of your life. Some of you might even have kids. And people in those two categories know that the technical event of getting a package and the marketing event of making the package available to people is a completely different thing. So release is a marketing thing. That's what we realized. Release is a marketing event. It's a marketing decision. It's something that is done at the point where marketing people want to achieve the biggest impact. You hide your packages nicely in a closet so when Christmas comes, children get surprised. Where we need to start doing the same stuff with software. We can decouple the technical risk of deploying to production through continuous delivery. And we can enable a marketing decision of releasing completely separately. Now, the key thing to unlock that. And that's a really, really interesting thing to think about here because it took us a while to figure out how to do this. But there's one technical problem with this. And once that's solved, everything works okay. And it's not an easy technical problem. It's a very difficult technical problem. That's why I suggested it's going to take five years. So the key technical thing here is to support multiple versions of software in production at the same time. So we can deploy one thing. The other thing can keep running. That means that if we start thinking about interrupting people's sessions, not a problem at all. They can keep running whatever version of software they're running at the moment. If we start thinking about surprising people and confusing people with UX, that's not a problem at all. We can still keep working on this testing, deploying. If we talk about interrupting marketing initiatives or disrupting stuff, that's not a problem at all because marketing people can decide when to do a release. So that's kind of the key thing to solve. So for us, the biggest challenge when we looked at this kind of collaboration tool, restructuring it for that was data. And I think data is going to be the biggest problem for most people because you can pretty much run multiple versions of APIs. You can run multiple versions of sites, websites, scripts. That's not a problem. Data tends to be the biggest problem. So our approach to this was we slowly started putting in data versions on pretty much every single data item. And the biggest challenge for us was we don't own the data at all. Most data sits on people's computers or on their Google Drive or their Dropbox. So as we upgrade versions, I don't even have access to all the data that's out there. And I can't even guarantee that all the data is going to be upgraded over a period of time. We can get hit by some data item that's over a year old because people just didn't need to use that. And we started putting kind of data flags, version flags on each data item and making sure that kind of newer versions of software can always automatically upgrade any previous version. That's a nice technical problem. It's a solvable technical problem. It's something that you can unit test. It's kind of something that is a technical problem. We also made sure that each kind of version of software deploying is just going to ignore any data items it doesn't recognize. It's not going to delete them. So pretty much at this point, we were able to have two users collaborating on the same document, running with completely different versions of software where the person on the latest version will have all the brand new shiny features and they would see that in their data. The person on the old version will see what they can see with their version, but they're not going to be interrupted. They're going to use everything they could. They will not notice anything that happened until they decide to kind of update or refresh the page. So in terms of this stuff, when we started doing it, then we also started time-stumping and data versioning all the scripts. So pretty much everything that goes online is time-stumped when it went online and put on a CDN. So what we've achieved with this is if you're a user of a site and you hit a website, you get an HTML page and that HTML page has time-stumped JavaScript, time-stumped CSS, every other resource time-stumped. It knows the data version. It knows the API version. So as it sends stuff to APIs, it sends the version and until you refresh the page, you have no idea what we are doing in the background because you are loading scripts that already exist. You can kind of have a cached page, live with the cached page as much as you want. We are kind of deleting these files once every six months. And the odd thing about this is once we started doing this, we started realizing really, really weird stuff because I originally thought we'll only need to keep two versions, the previous one and the one that we're running now, but then we realized lots of mobile networks all over the world cache HTML files. They don't cache JavaScript. So people would get all the HTML files for a week. I have no idea where from. And they would ask for JavaScripts that should have been inaccessible for a week. But because we're doing this stuff, those people could still use their software. And these were some kind of really weird errors that we were getting in our logs. We couldn't understand once we started doing this, they were no longer a problem. So this kind of started opening up some really, really interesting opportunities for us, but kind of the really important thing that I want you to remember from this. And it's a technical problem to be solved. It's not an easy problem, but I think continuous delivery without multi version is just irresponsible. People that say that they're doing continuous delivery and they can't run multiple versions of software in production at the same time are damaging their business. That's it. So if we are going to be professionals, we need to start restructuring our systems, restructuring data, restructuring APIs to be able to do multi version stuff. Now, once we started doing this, the whole kind of tall factory thing broke down. And a completely new world opened up because all of the sudden, we could take all those big, massive things that were impossible to slice in any meaningful way. And they were actually quite easy to slice and push out. So one thing that we started looking at is now we had this stuff we could actually slice by, slice by capacity. Because we could start rolling stuff out that supports up to 10 users at a time. It's perfectly fine that we have hundreds and hundreds of users on the system at the moment. I think at our busiest moment, we had about 600 users concurrently. It's perfectly fine to do something that only supports 10. Because we can run multiple versions at the same time and we can deploy a version that supports only 10 users to up to 10 users at a time. And then test that, see how that's going. And if it's going well, start kind of upgrading the stuff in the back end to support 50 users at a time, then support 100 users at a time. And when you start doing stuff, like lots of really interesting opportunities open up, for example, you don't have to use the database, you can use a file. You don't have to support, you can support one user at a time, if you want. And things like this kind of really, really weird and interesting stuff open up that are now a possibility to slice things differently. And we can actually push this stuff to production to measure the impact of that on one user. But hey, we can at least prove what's happening. So the other thing that we kind of were able to do now is start slicing things that are big, big UX changes in a way that's a lot more meaningful. For example, from Chrome version 26 to Chrome version 34, about four versions came up with broken canvas. That was a big problem for us because our key platform for visualization was canvas. And when Chrome breaks canvas, about 50% of our users pretty much complain to us. And we got a ton of angry emails all the time. And that's not something I can fix. I have no access to Chrome code. I wouldn't know even where to fix it. And generally kind of the Chrome guys were depending on where you look at it, either relatively good fixing it because it only took them a couple of days or really, really slow fixing it because it only took them a couple of days. But we had outages for 50% of our users for a couple of days where in some situations we could provide workarounds. Some situations, no way to do anything. Like one time we discovered that if you disable 3D acceleration, things work okay. So we sent emails to people. But then that's not really the best because they've disabled it globally. Some other changes just we couldn't fix. And we decided that kind of canvas is too unstable people are kind of still developing it. We're going to move to SVG. Now, this is one of those typical things where it's all or nothing because the value only comes where you can kill canvas completely. And if you're doing stuff gradually say you keep people both on canvas and SVG, that doesn't make sense at all. Because they're still going to be heard by canvas and your half baked SVG implementation is going to be ridiculous. So kind of no point at all until you have a big switch. But what we started thinking about is, hey, maybe we don't need to provide a whole implementation. Maybe we can look at getting 1% of our users on SVG and giving them an incentive to move to that. So we started looking at slicing by capacity and we did, oh, you know, our documents can have thousands and thousands and thousands of entries. The first thing we're going to do is we're going to do SVG that's not performance optimized. So we're going to support up to 100 elements. And we're not going to support all the features. We're going to support only these things. But what we're going to do is we're going to give people a nice incentive to choose this new stuff. So we deployed a very slimmed down version of SVG support that was performance wise bad that didn't have all the features, but it was enough for a sub segment of users to say, hey, this is good. I'm going to try this. And unlike PayPal that forced people to use the new business dashboard, what we did is we also introduced something that was new and funky. And we invited people to turn it on. So we gave people some new customizations and we said, look, you know, if you try this now, switch to it and you can get this fully customizable visuals. The performance is not good. You're not going to get all the features, blah, blah, blah, blah, blah, but try it out. It might work for you. And if it doesn't, you can always switch back. And by doing that, we're able to gradually kind of move people on. We were able to test that SVG is actually good. And during this kind of two months move to SVG, Chrome actually came up with a broken canvas. At this point, we were able to do about 50% of the users and just said, look, by now, we're just going to switch people to default SVG. If you're having problems, move back to canvas. But because Chrome is kind of borked, it might not work. And this is what Marty Kagan describes as gentle deployments in his book, Inspired. He talks about doing big exchanges by releasing something to a small portion of your users, inviting them to try it out and giving them the benefit to switch to the new thing while still allowing people to switch back to the old version if they have problems. This makes you exchanges zero risk because people intentionally switching to it, expecting consistency, so they're not going to be confused. They know what they're doing. They know why they're doing it. And if they have any problems, they can switch back. So that's a really nice way of kind of testing you exchanges because we can start bringing users gradually. We can start testing that those things make sense. And we can start doing technology refreshes that way. We can start moving people gradually to that. So kind of the key thing that I kind of like you to think about here is that kind of generally two things to remember from this are decouple deployment and release. Deployment is a technical event. Release is a marketing event. Although we were trained to think that those two things are the same, they're actually not. The tight coupling of deployment and release is a constraint of the old world. That's kind of the tall factory thing. And what we need to do is move away from that. And the kind of the right way to start doing that is to start supporting multi-version in production. And that's a big technical challenge for a lot of people, but it's a technical problem. It's something we can solve. And then we can reduce the risk of business changes, big UX refreshes, big technology refreshes, reducing kind of those big risks down to a technical problem. That's amazing. The second thing is this allows us now to start planning to give 2% of our users 100% of what they need rather than force feeding people with what they don't really want. And I think that's kind of the new world. That's where we start breaking the constraints of the old world because we can start doing that actively in production. And kind of when we start doing this amazing stuff start to happen. For example, there's a fantastic episode from Google in 2009 that is known as 41 Shades of Blue. This came up five years before the 50 Shades of Grace. So kind of, you know, they were there first and I think they could patent the whole thing. So at some point, Google's director of design had an idea to change the color of the hyperlinks in Google Ads on the homepage. And he proposed that they need to change this to this color because it's much more appealing to human eye. It's going to be great. It's going to be fantastic. The design is going to be much better. And because the designers were a bit pissed off at him for kind of previous stuff, they didn't just jump into doing this. They approached this as, okay, what we're going to do is we're going to decouple deployment and release. And we're going to scale this out so that marketing people can release it as they want. And by the way, because we don't trust that this is a good idea, what we're going to do is we're going to do the same thing with 40 other Shades of Blue at the same time. And we're going to let the marketing people decide on the release. We've done our technical part. Now other people can play with their stuff. And what they've done is kind of the marketing guys tested this on 1% of their users. And they proved that the color that the head of design suggested was actually performing much worse than the current one. And multiplied to 100% of the users and a year, the estimate was that the change in click rates would be about $200 million of damage. Now, instead of just saying, oh, you know, we got this requirement, we can roll it out quickly, we've rolled it out, it's nice, it's brilliant, we do continuous delivery, these guys decouple deployment and release. And as a result, they saved $200 million. The head of design actually quit as a result of that. So they saved $200 million plus a salary of an overpaid designer. And he went on to work for Twitter where they're still trying to figure out how to make money. So kind of that's the kind of stuff we can start doing if we do those two things. And they're not going to be easy, but they're incredibly powerful. So that's my message to you. Those are the two things to remember now. Kind of you had a good idea about the kind of three rules. And I think you said something. So you get two more books, I have two books here. The others, I don't want to make you feel bad, because you know, now you feel entitled. So I'll give you an electronic copy of this book. So if you go to this URL over the next month, you can download any copy of the book, it automatically reduces the price to zero. And a lot of the ideas that I've mentioned today are explained there in a bit more detail with some more references for research. So that's pretty much it. I hope I've challenged your thinking at least a bit. And I have about five minutes for questions, but then I have to go because I'll miss my plane. Does anybody have any questions that are kind of burning and you're not going to be able to sleep? Good. Okay, I'll go and catch my plane then. Thank you very much. Okay.
 Are we on? Hello? Yeah, good. Fantastic. So, okay, the idea for this talk is basically to try and  pack in one hour two ideas that if you think about those two ideas over the next five years, I  kind of guarantee that you will be able to become a lot more important in your company.  And although they're going to be very, very difficult to implement, if you persevere and do  them over the next five years, I will guarantee that you will turn IT from a cost center to a  value center in your company. So, kind of, that's not bad for a one hour of your time. And given  that kind of, I've messed up my air travel planning, I'll have to run away about 10 minutes early,  so it's not going to be even a whole hour. And from that perspective, we're going to look at  a couple of things that I think are going to happen in the near future to most people here.  And future, as always, is kind of elusive and uncertain. And although, you know, we don't  exactly know what's going to happen in the future, we can pretty much talk about some things with  certainty. For example, nobody knows when the world is exactly going to end, but pretty much  anybody who's done any development over the last five years knows that the apocalypse is going to  be caused by copy paste from Stack Overflow. So, some of those things are kind of more or less  certain. And I think a couple of patterns are emerging now that I'm seeing with lots of different  people that are going to be really, really interesting for the industry over the next five  years. And one of them is what's happening with continuous delivery. So Ian Cooper did a really  good talk yesterday. He talked about microservices and continuous delivery. And he said that kind of  as stuff gets faster and faster, people's appetite for risk increases, because continuous delivery  starts decreasing the risk of deployments and the risk of releases. And then people's appetite  increases quite a lot. And I think that kind of changes dynamics in a couple of really interesting  ways that software people don't necessarily see. So that's the first thing I want to talk about,  kind of how do we start spotting these wider patterns? So kind of the key message for today  is pretty much that the continuous delivery has serious side effects that software people don't  often consider. And by looking at these side effects, by trying to kind of use them for our  own purpose, we can open up new business opportunities, we can open up new markets, we can get a lot  more value out of continuous delivery. At the same time, if we decide to ignore these things,  and if we approach continuous delivery from a purely technical perspective, we can mess up  really, really, really badly. And I can tell you about this from my own kind of personal experience  I've kind of messed up pretty badly. The nice thing about coming to speak at the same conference  over several years is I can always come and talk about my new mistakes and people can laugh about  that. And I can feel like an alcoholic synonymous where every year I make new mistakes and every  year there's something new to talk about. So kind of the stuff I want to talk about is mistakes I've  made last year effectively that are in this area. So kind of lots of people, especially from our  industry, when they look at continuous delivery, they look at kind of faster deployments, they look  at the trend in going faster and faster and faster, they start thinking about, well, how can possibly  this be bad? Because, you know, fast is good, isn't it? And if you look outside of software,  there's a ton of examples where fast is not necessarily good. There's a ton of examples  where fast stuff is really, really bad for you. And in a lot of these cases, kind of we tend to  consider fast as good, but that's not necessarily true. For example, I worked with a TV station  a couple of years ago where their developers started doing continuous delivery and they were  moving from a cycle where they were no longer releasing every few months, they were releasing  every few weeks and then every few days and then a couple of times a day. And what the developers  thought they were giving the business is a shiny new toy that's really fast, really good, shiny new  toy that's really fast, really good, that's kind of fantastic. Unfortunately, what the business  users were seeing is this shiny new toy, but it's a completely different scenario where they were  trapped inside something that was kind of changing too frequently and they really didn't want to use  that. The problem here was that the technical platform was designed so that every time there's  an update, everybody who's currently logged onto the system had to log off and re-log in again.  And when they had an update every couple of months, that was no problem at all.  When they were updating several times a day, the content managers were pissed off all the time.  So, kind of this is something where two completely different opinions about continuous  delivery emerged in a company where the business people hated it, technical people loved it and  it's completely strange to kind of have something like that. So, one of the key reasons for this  is a loss of human control. Fast is good when there's a human element of control over that.  When we lose human control over that, when business users that are kind of content managers in this  case have no say over what gets updated and when, because stuff continuously gets pushed to them,  we've lost control. And loss of control is something that kind of sci-fi writers have been  warning us about for decades now. You have lots of stuff where, you know, robots emerge and take  over humanity and kill everybody. And if you want to go for a really bad scenario, then you have  robots that address to look like humans. If you want to go for a really, really, really, really,  really bad scenario, then you start thinking about robots that address like humans, but they look  harmless. So, that's kind of one of the biggest problems we have now is I think continuous  delivery is reaching this point where we really need to start thinking about can we control it,  or are we losing control over that. And again, going back to sci-fi, Issa Kasimov solved this  problem in 1948, and he proposed the three rules of kind of robotics to make sure that evil machines  don't take over the world and rule the world. So, I have three copies of my book to give away to  people who can remember at least one of these rules. Don't come a human, okay? It was the same,  okay? So, no. Well, okay, but that's kind of don't come another human. So, that's  so you get one.  Do not bring harm to yourself. Well, kind of, I'll give you that. I'll give you that. That's kind  of the third one where they have to continue their existence or protect their existence  unless it's conflicting with rule number one. Okay, so, kind of the three rules that Issa Kasimov  declared, the first of all, don't harm a human. The second one is obey everything humans tell you,  unless it's conflicting rule number one, and the third one is kind of try and protect yourself  unless kind of it's conflicting rule number one, rule number two. So, these three things are very,  very important. We'll come back to that because I think that's part of the secret how we can start  unlocking the real power of continuous delivery. Now, kind of in terms of what's been happening in  the industry, especially over the last five or six years, I think we're getting to a point where  the technical part of continuous delivery is kind of solved. But that's opening up some really,  really interesting new possibilities. And I think this is very close to what happened when factories  were first electrified. Before electric motors, what most factories used to have is they used to  have a boiler room in the basement, and they were built to be very tall because steam goes up. That's  how you harness steam. So, they would put a pipe there and then they would put lots of stuff on  each floor to be powered by the same pipe. Now, when the first electric motors came in, people got  something that was safer than steam, they got something that was more powerful, that was cheaper,  but they used the same paradigms for that. They would put the electric motor in the basement,  and they would power the whole thing. And it took 10 years for the industry to realize that the  constraints of the old world, the steam going only up, have been solved and they no longer exist.  And then they could kind of come up with lots and lots of interesting solutions for other problems.  For example, when a factory is very tall, you have to solve a problem of transporting things up and  down. And because that's no longer a constraint, people started building very, very flat factories  where it's very easy to transport stuff because electricity goes wherever the cables go. And I  think that's kind of the situation we are in now with continuous delivery, where we can start  dismantling stuff we built around the constraints of the old world because they no longer exist.  For example, in January last year, the American National Transport Safety Organization that  regulates their kind of transport companies issued two product recalls, both related to potential  fire hazards. One was for General Motors cars, and another one was for Tesla cars. And kind of,  for the Tesla car, they issued a recall that was going to affect about 20,000 vehicles,  and they were complaining about how a plug in the charger is a potential fire hazard.  Now, General Motors had to bring all their cars back to a repair shop to kind of fix them,  to send them back. Tesla said, no problem, we're going to do a software update, and it's going  to happen overnight. So pretty much when people woke up next morning, this problem was solved.  Now, this is an example of what I mean by we can start dismantling the constraints of the old world,  because when you have a car that can self update over the internet to resolve a hardware problem,  and that doesn't cost anybody almost anything. People don't have to drive their cars into a  repair shop. Tesla doesn't have to employ people to fix it. This is now a huge, huge,  huge competitive advantage where kind of General Motors has to fix cars, these guys issue a software  patch. And this is the flat factory thing, this is happening now. And at the same time,  continuous delivery starts creating some very, very unusual feedbacks on other parts of the  business as well. For example, about two years ago, I started working on a collaboration tool  with some friends, and we started doing stuff the way we're used to doing it. So we started with  continuous delivery from day one, we were packaging stuff, releasing them very quickly. And this is  our user growth graph over the last two years. Watching spot is somewhere towards the beginning,  there's a massive spike. Then there's another kind of spike in another spike over there,  very, very close to the beginning. And then for the next two years, no spikes at all.  What happened there is we appeared on Hacker News on the front page, then what happened there is  we appeared on Reddit on the front page. And then what happened there is the official version one  was reviewed by PC World. What happened for the other one and a half years is we tried as much  as we could to get anybody to write about us and failed. Although we did a ton of really,  really interesting things, what we learned is that when you do continuous delivery,  you take the drama out of the process. There's nothing newsworthy to write about. There's kind  of nothing big to announce. There's no marketing opportunities anymore and kind of our marketing  strategy that was to get blogs and newspapers to write about us was completely destroyed by the  fact that we took the drama out of delivery. So this is a really interesting aspect that I never  expected because previously when we did stuff like that, you'd release every six months or every  year, you announced version 3.0, people write about that because journalists like to kind of  write about something that's newsworthy. Now when you say, oh, we've updated this feature,  we've updated that feature, there's nothing, nothing to come out of that. So kind of,  although we perceived continuous delivery as a purely technical thing, it had a massive side  effect on our marketing. Now that's not necessarily horribly bad. It's just that it prevented us from  doing one marketing thing. We had to learn about that the hard way. And we can say, oh, I'm stupid  because I make mistakes every year and I'm kind of coming here to talk to you about this. But other  companies do this as well. And they get very, very surprised by the marketing side effects of  continuous delivery because that's not a purely technical thing. For example, I'm going back to  Tesla in October last year. Tesla announced a fantastic new feature that they're starting to  put in their cars. And kind of Tesla with the whole continuous delivery model is trying to  challenge this perception of models for cars. Ford, General Motors, Mercedes, they have models,  Tesla kind of has a model S and just kind of ships updates. So they start announcing that,  you know, from tomorrow morning, all model S cars that come out will have a front facing radar,  12 sonar sensors, several cameras, and they'll have an autopilot that's going to automatically  change lanes. They'll have a digital break that's going to spot when there's a dangerous situation  and kind of prevent stuff like that. So this is something that's amazing. It's kind of a fantastic  safety feature. It's a fantastic feature anyway. And everybody was very excited about that.  Newspapers wrote about how shit scared about this because when you think about Tesla doing  automatic navigation, radar stuff, sensor stuff, you kind of know what the end game is going to be.  So where kind of, I wasn't the only one that kind of felt negative about this. There was a guy  called Richard Wolpert in the States that woke up one day, read the news about this, and he just  felt that the world was unfair. Everything went wrong that day because he just bought a model S car  that didn't have all these nice shiny features. And here's Tesla announcing that from tomorrow  everybody else is going to get the radar, a sonar, and I don't know why they need a sonar. I guess  they're preparing a car to go underwater. But kind of, you know, 12 sonars, two radars, cameras,  automatic navigation, things like that. Richard felt that the world was just unfair. So what do  people do when they feel the world is unfair? They start bitching about it online. And he started  a petition to kind of force Tesla to get autopilot radars and sonars retrofitted into all the old  models. And 981, 948 people signed up for this petition to kind of start forcing Tesla to retrofit  stuff because people felt entitled. As continuous delivery teaches people that we can now ship stuff  easily, update stuff, users start feeling more entitled. Now, I've never heard of anybody going  to Mercedes or to Ford and saying, Oh, you have a new model. I bought the car five years ago.  I want everything that the new car has for free. But Tesla taught their users to expect more.  And, you know, people all over the world signed up for this. Where, you know, even in Norway,  there was a guy in Norway called Jochen, who said, Hey, because it's been said, I want this stuff,  I'm entitled to this stuff. And this is because kind of when we as we start doing continuous  delivery, our stakeholders, our users start expecting stuff that might be completely unrealistic.  And, you know, installing radars and sonars and stuff like that, you know, that's not a software  update thing. That's kind of Star Trek replicator stuff. That's not something you can do very,  very easily. But again, these people felt entitled. So regardless of whether that's possible or not  possible, disappointing users is never good. Making users angry enough to start an online petition  against you is never good. Tesla is still in an early adopter stage. And at that point, they were  selling something like one and a half thousand cars a month. So if you have 1000 people pissed  off enough to sign an online petition, and you're selling one and a half thousand cars a month,  that's not good. So kind of this is where I think we need to start thinking about this slightly  differently. And especially when kind of users are concerned, continuous delivery is typically  something that technical people approach as a way to reduce technical risk. And users typically get  at the wrong end of the stick. So if you're doing continuous delivery, please, please, please, please,  consider your poor users. For example, I tend to travel a lot for conferences. And I was in Australia  last year. Australia, unlike kind of normal countries, has stupidly expensive internet.  I'm talking about, say, $50 for three gigabytes. And they sell it by gigabyte, they don't sell it  by time based access and things like that. So as we were there, the conference organizers gave us  these small kind of my five devices to use so that we don't pay the hotels. And I kind of like to  fiddle with my slides until the very last moment I was changing the slides as people were coming  into this room now. So about two hours before my talk in Australia, I finally sat down to do my  slides. And normally that's not a problem at all. You go to Google Images, you find Patrick Stewart,  you put it on, and that's perfect. That day, the world seemed unfair. None of the browsers seem  to work. Everything seemed to crash all the time. My ping was 10 seconds to anything. And I couldn't  figure out what was going on. After about half an hour of panicking, restarting the computer,  trying to figure out whether Apple's network demon is a fault or whatever is going on, I've  realized that my iPad decided on its own to update iOS and kind of took out the complete pipe I had.  And not only that, but kind of spent all the bandwidth I had left. So I had to pay $50 for two  more gigabytes. Then I had to find slides in a panic mode and kind of do with a half-finished  presentation. So fair enough, that's my fault. It's my fault for not disabling automatic updates.  But at the same time, Apple had an important update. And the update is never, ever, ever  going to be as important as the work I have to do now. When people have an important security  update, that's important for the company doing the update. That's basically saying, we cocked up,  we're embarrassed, and now we want to update your system. If I'm doing something at that moment,  that's not important at all. That's very low on the scale of importance. Likewise, for the content  managers at TV Station, the technical update that was done for some other users somewhere  half way around the world was completely unimportant to them. What was important is they were half  way through writing a story, and now they have to log out and re-log in again. And I think we need to  start considering that if we're not going to piece of users, if you really, really, really, really,  really, really, really want a piece of users, then what you should do is deploy UX changes partially.  And do continuous delivery on the UX. For example, PayPal did this to me about six months ago.  They were starting to deploy the new business dashboard. Whenever you hear news somewhere,  run away, run to the hills. So I woke up one day and there were a few thousand dollars missing  from my company PayPal account. Now, that's never good. And because several people have access to  the account, I didn't know whether somebody took the money out or whether they were going to  go to where the kind of we got defrauded or something like that. So in panic mode, I started  downloading transactions, figuring out what was going on, adding things up, but then they just  didn't add up. If I added up the transactions, it looked fine. If I looked at the balance in the  business dashboard, several thousand dollars were missing. So kind of I noticed a link on top.  This is switched to classic and I put my testers hat down and I thought somebody must have caught  something up. So I switched to classic and there was the money. So what effectively they did is we  had multiple currencies in our accounts. And because the primary account was British pounds,  they kind of deployed the UX changes gradually and they thought, well, we're just going to do it  first for people with one account. And they were only showing the British balance. Now,  I could put my testers hat on and I could notice the switch to classic thing, but I can bet you  lots and lots of people that they had a really bad day and the call center supported people had a  really bad day as well. So kind of that's why we're, you know, one of the biggest problems when we  talk about continuous delivery, although continuous delivery is incredibly empowering. It opens up new  business opportunities. It can kind of help you be Tesla, not general motors and stuff like that.  When you go and talk to people about doing big stuff, they're really, really scared.  An insurance company I worked with, when they started moving to continuous delivery, the  business users thought, this is really brilliant. This is great. We can do small stuff easily,  but our technology lost the capability of doing any big changes anymore because they just fight  anything that's big. When you go and talk to UX people and you say, okay, we're going to start  updating one page and we're going to update another, they start running for the hills because  if we start looking at this from a development perspective, we think what we're doing is Amazon  Prime. We think we're shipping stuff well. We think it's all fine. It's all brilliant.  But when you look at it from a user perspective, from a UX perspective, what those people feel  like is they're being force fed. And there's kind of a massive difference between continuous  delivery this way and continuous delivery that way. So one of the most important things we need  to start thinking about, especially kind of doing big, big changes and not losing the capability  of doing big stuff is not doing this stuff over here. So kind of, I've almost always,  when we start talking about continuous delivery to business people in terms of reducing technical  risk, they love it. Then you go and talk to kind of how you're going to do a big technology refresh  or how you're going to risk in the website, how you're going to rebrand something when you need,  and people start fighting you with everything they can. UX designers kind of complain that you  can't go and confuse users because if you start rebranding one part of the website,  the other part is going to be completely confused. And that's what happened to us. I mean, I can't  complain about that. And especially when you start thinking about slicing things, it's almost  impossible to slice continuous delivery of UX in any sensible way. People will always be confused  and it's always going to be stupid. So we can either not do continuous delivery or we can do it in a  way that confuses people. And that's where kind of, this is really, really a big challenge. So  one of the most important things to unlock the potential of this over the next five years  is going to look at how do we not lose human control over this? How do we not go for false  feeding? How do we go for Amazon Prime? And how do we, if we have to do this, because we'll have to  do it. But how do we at least keep human control over that? So we do it in a sensible way because  there are sensible ways of doing this. So kind of going back to the three rules of robotics,  I think kind of what I'd like to rewrite this, so we're not going to talk about robots anymore,  we'll talk about continuous delivery. And the first rule that talks about may not injure a human.  Fair enough, if people are doing kind of medical devices, I hope you already have this in your  testing. But when we talk about continuous delivery, particularly when we talk about UX, I think  the biggest problem here is confusion. So the first rule of continuous delivery that we really have  to put in place is this can't confuse users. Now, the second rule talks about obeying orders for  humans, unless kind of order one, and I'm going to kind of skip the unless order, unless rule one.  I think kind of this thing may must obey users when we think about what people actually want in  terms of continuous delivery. Most often, they don't want it. If I'm in the middle of something,  I do not want the ground to move. I want the stuff to stay the same. And I want this stuff,  if it has to happen, to happen without me noticing at all. So the second rule of continuous delivery,  if we're going to do it right, I'm going to replace must obey orders from must not interrupt users'  work or sessions, because that's the really, really important thing. If that happens, I don't care  about. Now, the third one is interesting kind of must protect its own existence. And where continuous  delivery fails to protect its own existence, if it starts fighting the business. If there's a business  reason why we want to risk in the website, or there's a business reason why we want to do  technology refresh, we want to move from Oracle to in-memory data grids, we want to do a huge  change that is going to involve rewriting everything to work on a mobile and things like  that. Continuous delivery cannot fight the business there. If we start saying, oh, you know, that's  possible, but only if we start moving people and for a while everything's going to be messed up,  that's not going to work. And one of the most important things there is, I think, continuous  delivery has a huge potential to disrupt marketing. That's what we felt on our own scale. That's what  Tesla felt. So I think kind of continuous delivery rule number three in terms of protecting its own  existence must be not to disrupt or prevent marketing initiatives. And if we can do this kind of stuff,  then we can put human control around all this robotic mechanism and we can actually benefit  quite a lot from the whole thing. So when we started looking at these three things for the  collaboration tool I mentioned, we realized very, very quickly that we had a fundamental  misunderstanding of two really important concepts. In our mind, deployment and release were the same  thing. And for most software people, and it's likely that most of in the audience are now kind of  scratching your heads, deployment and release are the same thing. But kind of we realized that's  completely wrong. And that's one of those constraints of the old world. That's a tall factory. And if  we can move away from that because those constraints no longer apply, then we can get really, really,  really, really interesting benefits out of the whole thing. So the first thing that I'd really  like you to kind of remember from this is that deployment isn't and shouldn't be the same thing  as a release. Deployment is a technical event of pushing software to production. And I'm really  thinking production here. I don't care about deployment to staging. I don't care about deployment  to development environments. That's all nice and fine. But deployment is deployment to production.  That's a technical event of moving code to production. Now, most of you, I assume, have  been kids at some point of your life. Some of you might even have kids. And people in those two  categories know that the technical event of getting a package and the marketing event of  making the package available to people is a completely different thing. So release is a  marketing thing. That's what we realized. Release is a marketing event. It's a marketing decision.  It's something that is done at the point where marketing people want to achieve the biggest  impact. You hide your packages nicely in a closet so when Christmas comes, children get surprised.  Where we need to start doing the same stuff with software. We can decouple the technical risk  of deploying to production through continuous delivery. And we can enable a marketing decision  of releasing completely separately. Now, the key thing to unlock that. And that's a really,  really interesting thing to think about here because it took us a while to figure out how to  do this. But there's one technical problem with this. And once that's solved, everything works  okay. And it's not an easy technical problem. It's a very difficult technical problem. That's  why I suggested it's going to take five years. So the key technical thing here is to support  multiple versions of software in production at the same time. So we can deploy one thing. The other  thing can keep running. That means that if we start thinking about interrupting people's sessions,  not a problem at all. They can keep running whatever version of software they're running at  the moment. If we start thinking about surprising people and confusing people with UX, that's not  a problem at all. We can still keep working on this testing, deploying. If we talk about  interrupting marketing initiatives or disrupting stuff, that's not a problem at all because  marketing people can decide when to do a release. So that's kind of the key thing to solve. So for  us, the biggest challenge when we looked at this kind of collaboration tool, restructuring it for  that was data. And I think data is going to be the biggest problem for most people because you can  pretty much run multiple versions of APIs. You can run multiple versions of sites, websites,  scripts. That's not a problem. Data tends to be the biggest problem. So our approach to this was  we slowly started putting in data versions on pretty much every single data item. And the biggest  challenge for us was we don't own the data at all. Most data sits on people's computers or on their  Google Drive or their Dropbox. So as we upgrade versions, I don't even have access to all the  data that's out there. And I can't even guarantee that all the data is going to be upgraded over a  period of time. We can get hit by some data item that's over a year old because people just didn't  need to use that. And we started putting kind of data flags, version flags on each data item  and making sure that kind of newer versions of software can always automatically upgrade any  previous version. That's a nice technical problem. It's a solvable technical problem. It's something  that you can unit test. It's kind of something that is a technical problem. We also made sure that  each kind of version of software deploying is just going to ignore any data items it doesn't  recognize. It's not going to delete them. So pretty much at this point, we were able to have two  users collaborating on the same document, running with completely different versions of software  where the person on the latest version will have all the brand new shiny features and they would  see that in their data. The person on the old version will see what they can see with their version,  but they're not going to be interrupted. They're going to use everything they could.  They will not notice anything that happened until they decide to kind of update or refresh the page.  So in terms of this stuff, when we started doing it, then we also started time-stumping and data  versioning all the scripts. So pretty much everything that goes online is time-stumped when  it went online and put on a CDN. So what we've achieved with this is if you're a user of a site  and you hit a website, you get an HTML page and that HTML page has time-stumped JavaScript,  time-stumped CSS, every other resource time-stumped. It knows the data version. It knows the API  version. So as it sends stuff to APIs, it sends the version and until you refresh the page,  you have no idea what we are doing in the background because you are loading scripts that already exist.  You can kind of have a cached page, live with the cached page as much as you want. We are kind of  deleting these files once every six months. And the odd thing about this is once we started doing  this, we started realizing really, really weird stuff because I originally thought we'll only need  to keep two versions, the previous one and the one that we're running now, but then we realized lots  of mobile networks all over the world cache HTML files. They don't cache JavaScript. So people would  get all the HTML files for a week. I have no idea where from. And they would ask for JavaScripts that  should have been inaccessible for a week. But because we're doing this stuff, those people could  still use their software. And these were some kind of really weird errors that we were getting in our  logs. We couldn't understand once we started doing this, they were no longer a problem. So this kind  of started opening up some really, really interesting opportunities for us, but kind of the really  important thing that I want you to remember from this. And it's a technical problem to be solved.  It's not an easy problem, but I think continuous delivery without multi version is just irresponsible.  People that say that they're doing continuous delivery and they can't run multiple versions of  software in production at the same time are damaging their business. That's it. So if we are going  to be professionals, we need to start restructuring our systems, restructuring data, restructuring  APIs to be able to do multi version stuff. Now, once we started doing this, the whole kind of  tall factory thing broke down. And a completely new world opened up because all of the sudden,  we could take all those big, massive things that were impossible to slice in any meaningful way.  And they were actually quite easy to slice and push out. So one thing that we started  looking at is now we had this stuff we could actually slice by, slice by capacity.  Because we could start rolling stuff out that supports up to 10 users at a time.  It's perfectly fine that we have hundreds and hundreds of users on the system at the moment.  I think at our busiest moment, we had about 600 users concurrently. It's perfectly fine to do  something that only supports 10. Because we can run multiple versions at the same time and we can  deploy a version that supports only 10 users to up to 10 users at a time. And then test that,  see how that's going. And if it's going well, start kind of upgrading the stuff in the back end  to support 50 users at a time, then support 100 users at a time. And when you start doing stuff,  like lots of really interesting opportunities open up, for example, you don't have to use the  database, you can use a file. You don't have to support, you can support one user at a time,  if you want. And things like this kind of really, really weird and interesting stuff open up that  are now a possibility to slice things differently. And we can actually push this stuff to production  to measure the impact of that on one user. But hey, we can at least prove what's happening.  So the other thing that we kind of were able to do now is start slicing things that are big,  big UX changes in a way that's a lot more meaningful. For example, from Chrome version 26 to Chrome version  34, about four versions came up with broken canvas. That was a big problem for us because  our key platform for visualization was canvas. And when Chrome breaks canvas, about 50% of our  users pretty much complain to us. And we got a ton of angry emails all the time. And that's not  something I can fix. I have no access to Chrome code. I wouldn't know even where to fix it.  And generally kind of the Chrome guys were depending on where you look at it, either  relatively good fixing it because it only took them a couple of days or really, really slow  fixing it because it only took them a couple of days. But we had outages for 50% of our users  for a couple of days where in some situations we could provide workarounds. Some situations,  no way to do anything. Like one time we discovered that if you disable 3D acceleration, things work  okay. So we sent emails to people. But then that's not really the best because they've disabled it  globally. Some other changes just we couldn't fix. And we decided that kind of canvas is too unstable  people are kind of still developing it. We're going to move to SVG. Now, this is one of those  typical things where it's all or nothing because the value only comes where you can kill canvas  completely. And if you're doing stuff gradually say you keep people both on canvas and SVG,  that doesn't make sense at all. Because they're still going to be heard by canvas and your half  baked SVG implementation is going to be ridiculous. So kind of no point at all until you have a big  switch. But what we started thinking about is, hey, maybe we don't need to provide a whole  implementation. Maybe we can look at getting 1% of our users on SVG and giving them an incentive  to move to that. So we started looking at slicing by capacity and we did, oh, you know, our documents  can have thousands and thousands and thousands of entries. The first thing we're going to do is we're  going to do SVG that's not performance optimized. So we're going to support up to 100 elements.  And we're not going to support all the features. We're going to support only these things. But  what we're going to do is we're going to give people a nice incentive to choose this new stuff.  So we deployed a very slimmed down version of SVG support that was performance wise bad  that didn't have all the features, but it was enough for a sub segment of users to say, hey,  this is good. I'm going to try this. And unlike PayPal that forced people to use the new business  dashboard, what we did is we also introduced something that was new and funky. And we invited  people to turn it on. So we gave people some new customizations and we said, look, you know, if you  try this now, switch to it and you can get this fully customizable visuals. The performance is  not good. You're not going to get all the features, blah, blah, blah, blah, blah, but try it out.  It might work for you. And if it doesn't, you can always switch back. And by doing that, we're  able to gradually kind of move people on. We were able to test that SVG is actually good. And  during this kind of two months move to SVG, Chrome actually came up with a broken canvas. At this  point, we were able to do about 50% of the users and just said, look, by now, we're just going to  switch people to default SVG. If you're having problems, move back to canvas. But because Chrome  is kind of borked, it might not work. And this is what Marty Kagan describes as gentle deployments  in his book, Inspired. He talks about doing big exchanges by releasing something to a small  portion of your users, inviting them to try it out and giving them the benefit to switch to the  new thing while still allowing people to switch back to the old version if they have problems.  This makes you exchanges zero risk because people intentionally switching to it, expecting  consistency, so they're not going to be confused. They know what they're doing. They know why they're  doing it. And if they have any problems, they can switch back. So that's a really nice way of  kind of testing you exchanges because we can start bringing users gradually. We can start testing  that those things make sense. And we can start doing technology refreshes that way. We can start  moving people gradually to that. So kind of the key thing that I kind of like you to think about  here is that kind of generally two things to remember from this are decouple deployment and  release. Deployment is a technical event. Release is a marketing event. Although we were trained to  think that those two things are the same, they're actually not. The tight coupling of deployment  and release is a constraint of the old world. That's kind of the tall factory thing. And what we  need to do is move away from that. And the kind of the right way to start doing that is to start  supporting multi-version in production. And that's a big technical challenge for a lot of people,  but it's a technical problem. It's something we can solve. And then we can reduce the risk of  business changes, big UX refreshes, big technology refreshes, reducing kind of those big risks  down to a technical problem. That's amazing. The second thing is this allows us now to start  planning to give 2% of our users 100% of what they need rather than force feeding people with  what they don't really want. And I think that's kind of the new world. That's where we start  breaking the constraints of the old world because we can start doing that actively in production.  And kind of when we start doing this amazing stuff start to happen. For example, there's a  fantastic episode from Google in 2009 that is known as 41 Shades of Blue. This came up  five years before the 50 Shades of Grace. So kind of, you know, they were there first and I think  they could patent the whole thing. So at some point, Google's director of design had an idea  to change the color of the hyperlinks in Google Ads on the homepage. And he proposed that they  need to change this to this color because it's much more appealing to human eye. It's going to be  great. It's going to be fantastic. The design is going to be much better. And because the designers  were a bit pissed off at him for kind of previous stuff, they didn't just jump into doing this.  They approached this as, okay, what we're going to do is we're going to decouple deployment and  release. And we're going to scale this out so that marketing people can release it as they want.  And by the way, because we don't trust that this is a good idea, what we're going to do is we're  going to do the same thing with 40 other Shades of Blue at the same time. And we're going to let  the marketing people decide on the release. We've done our technical part. Now other people can  play with their stuff. And what they've done is kind of the marketing guys tested this on 1% of  their users. And they proved that the color that the head of design suggested was actually performing  much worse than the current one. And multiplied to 100% of the users and a year, the estimate was  that the change in click rates would be about $200 million of damage. Now, instead of just saying,  oh, you know, we got this requirement, we can roll it out quickly, we've rolled it out, it's nice,  it's brilliant, we do continuous delivery, these guys decouple deployment and release. And as a  result, they saved $200 million. The head of design actually quit as a result of that. So they  saved $200 million plus a salary of an overpaid designer. And he went on to work for Twitter  where they're still trying to figure out how to make money. So kind of that's the kind of stuff  we can start doing if we do those two things. And they're not going to be easy, but they're  incredibly powerful. So that's my message to you. Those are the two things to remember now.  Kind of you had a good idea about the kind of three rules. And I think you said something. So  you get two more books, I have two books here. The others, I don't want to make you feel bad,  because you know, now you feel entitled. So I'll give you an electronic copy of this book.  So if you go to this URL over the next month, you can download any copy of the book, it automatically  reduces the price to zero. And a lot of the ideas that I've mentioned today are explained there in a  bit more detail with some more references for research. So that's pretty much it. I hope I've  challenged your thinking at least a bit. And I have about five minutes for questions, but then I  have to go because I'll miss my plane. Does anybody have any questions that are kind of burning  and you're not going to be able to sleep? Good. Okay, I'll go and catch my plane then. Thank you  very much.  Okay.
How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_0_3803.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_3804_3851.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_3852_4199.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_4200_4583.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_4584_4631.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_4632_5867.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_5868_6683.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_6684_6947.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_6948_8891.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_8892_9071.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_9072_9311.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_9312_9875.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_9876_11387.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_11388_15419.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_15420_18011.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_18012_18467.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_18468_18515.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_18516_21995.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_21996_22043.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_22044_22511.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_22512_22595.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_22596_22667.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_22668_22715.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_22716_22835.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_22836_22967.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_22968_24503.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_24504_25667.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_25668_25715.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_25716_26411.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_26412_35207.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_35208_35663.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_35664_37211.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_37212_38531.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_38532_39299.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_39300_40403.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_40404_42011.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_42012_43919.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_43920_44267.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_44268_45191.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_45192_48623.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_48624_51023.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_51024_53627.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_53628_54551.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_54552_54611.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_54612_55703.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_55704_60563.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_60564_61175.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_61176_63227.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_63228_63395.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_63396_63863.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_63864_65759.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_65760_65855.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_65856_65939.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_65940_66059.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_66060_66239.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_66240_66359.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_66360_66407.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_66408_66455.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_66456_66611.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_66612_66755.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_66756_66863.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_66864_66911.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_66912_67115.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_67116_69839.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_69840_70943.png How_to_turn_software_into_competitive_business_advantage_-_Gojko_Adzic_ORIGINAL_DOWNLOAD_70944_74256.png
Are we on? Hello? Yeah, good. Fantastic. So, okay, the idea for this talk is basically to try and pack in one hour two ideas that if you think about those two ideas over the next five years, I kind of guarantee that you will be able to become a lot more important in your company. And although they're going to be very, very difficult to implement, if you persevere and do them over the next five years, I will guarantee that you will turn IT from a cost center to a value center in your company. So, kind of, that's not bad for a one hour of your time. And given that kind of, I've messed up my air travel planning, I'll have to run away about 10 minutes early, so it's not going to be even a whole hour. And from that perspective, we're going to look at a couple of things that I think are going to happen in the near future to most people here. And future, as always, is kind of elusive and uncertain. And although, you know, we don't exactly know what's going to happen in the future, we can pretty much talk about some things with certainty. For example, nobody knows when the world is exactly going to end, but pretty much anybody who's done any development over the last five years knows that the apocalypse is going to be caused by copy paste from Stack Overflow. So, some of those things are kind of more or less certain. And I think a couple of patterns are emerging now that I'm seeing with lots of different people that are going to be really, really interesting for the industry over the next five years. And one of them is what's happening with continuous delivery. So Ian Cooper did a really good talk yesterday. He talked about microservices and continuous delivery. And he said that kind of as stuff gets faster and faster, people's appetite for risk increases, because continuous delivery starts decreasing the risk of deployments and the risk of releases. And then people's appetite increases quite a lot. And I think that kind of changes dynamics in a couple of really interesting ways that software people don't necessarily see. So that's the first thing I want to talk about, kind of how do we start spotting these wider patterns? So kind of the key message for today is pretty much that the continuous delivery has serious side effects that software people don't often consider. And by looking at these side effects, by trying to kind of use them for our own purpose, we can open up new business opportunities, we can open up new markets, we can get a lot more value out of continuous delivery. At the same time, if we decide to ignore these things, and if we approach continuous delivery from a purely technical perspective, we can mess up really, really, really badly. And I can tell you about this from my own kind of personal experience I've kind of messed up pretty badly. The nice thing about coming to speak at the same conference over several years is I can always come and talk about my new mistakes and people can laugh about that. And I can feel like an alcoholic synonymous where every year I make new mistakes and every year there's something new to talk about. So kind of the stuff I want to talk about is mistakes I've made last year effectively that are in this area. So kind of lots of people, especially from our industry, when they look at continuous delivery, they look at kind of faster deployments, they look at the trend in going faster and faster and faster, they start thinking about, well, how can possibly this be bad? Because, you know, fast is good, isn't it? And if you look outside of software, there's a ton of examples where fast is not necessarily good. There's a ton of examples where fast stuff is really, really bad for you. And in a lot of these cases, kind of we tend to consider fast as good, but that's not necessarily true. For example, I worked with a TV station a couple of years ago where their developers started doing continuous delivery and they were moving from a cycle where they were no longer releasing every few months, they were releasing every few weeks and then every few days and then a couple of times a day. And what the developers thought they were giving the business is a shiny new toy that's really fast, really good, shiny new toy that's really fast, really good, that's kind of fantastic. Unfortunately, what the business users were seeing is this shiny new toy, but it's a completely different scenario where they were trapped inside something that was kind of changing too frequently and they really didn't want to use that. The problem here was that the technical platform was designed so that every time there's an update, everybody who's currently logged onto the system had to log off and re-log in again. And when they had an update every couple of months, that was no problem at all. When they were updating several times a day, the content managers were pissed off all the time. So, kind of this is something where two completely different opinions about continuous delivery emerged in a company where the business people hated it, technical people loved it and it's completely strange to kind of have something like that. So, one of the key reasons for this is a loss of human control. Fast is good when there's a human element of control over that. When we lose human control over that, when business users that are kind of content managers in this case have no say over what gets updated and when, because stuff continuously gets pushed to them, we've lost control. And loss of control is something that kind of sci-fi writers have been warning us about for decades now. You have lots of stuff where, you know, robots emerge and take over humanity and kill everybody. And if you want to go for a really bad scenario, then you have robots that address to look like humans. If you want to go for a really, really, really, really, really bad scenario, then you start thinking about robots that address like humans, but they look harmless. So, that's kind of one of the biggest problems we have now is I think continuous delivery is reaching this point where we really need to start thinking about can we control it, or are we losing control over that. And again, going back to sci-fi, Issa Kasimov solved this problem in 1948, and he proposed the three rules of kind of robotics to make sure that evil machines don't take over the world and rule the world. So, I have three copies of my book to give away to people who can remember at least one of these rules. Don't come a human, okay? It was the same, okay? So, no. Well, okay, but that's kind of don't come another human. So, that's so you get one. Do not bring harm to yourself. Well, kind of, I'll give you that. I'll give you that. That's kind of the third one where they have to continue their existence or protect their existence unless it's conflicting with rule number one. Okay, so, kind of the three rules that Issa Kasimov declared, the first of all, don't harm a human. The second one is obey everything humans tell you, unless it's conflicting rule number one, and the third one is kind of try and protect yourself unless kind of it's conflicting rule number one, rule number two. So, these three things are very, very important. We'll come back to that because I think that's part of the secret how we can start unlocking the real power of continuous delivery. Now, kind of in terms of what's been happening in the industry, especially over the last five or six years, I think we're getting to a point where the technical part of continuous delivery is kind of solved. But that's opening up some really, really interesting new possibilities. And I think this is very close to what happened when factories were first electrified. Before electric motors, what most factories used to have is they used to have a boiler room in the basement, and they were built to be very tall because steam goes up. That's how you harness steam. So, they would put a pipe there and then they would put lots of stuff on each floor to be powered by the same pipe. Now, when the first electric motors came in, people got something that was safer than steam, they got something that was more powerful, that was cheaper, but they used the same paradigms for that. They would put the electric motor in the basement, and they would power the whole thing. And it took 10 years for the industry to realize that the constraints of the old world, the steam going only up, have been solved and they no longer exist. And then they could kind of come up with lots and lots of interesting solutions for other problems. For example, when a factory is very tall, you have to solve a problem of transporting things up and down. And because that's no longer a constraint, people started building very, very flat factories where it's very easy to transport stuff because electricity goes wherever the cables go. And I think that's kind of the situation we are in now with continuous delivery, where we can start dismantling stuff we built around the constraints of the old world because they no longer exist. For example, in January last year, the American National Transport Safety Organization that regulates their kind of transport companies issued two product recalls, both related to potential fire hazards. One was for General Motors cars, and another one was for Tesla cars. And kind of, for the Tesla car, they issued a recall that was going to affect about 20,000 vehicles, and they were complaining about how a plug in the charger is a potential fire hazard. Now, General Motors had to bring all their cars back to a repair shop to kind of fix them, to send them back. Tesla said, no problem, we're going to do a software update, and it's going to happen overnight. So pretty much when people woke up next morning, this problem was solved. Now, this is an example of what I mean by we can start dismantling the constraints of the old world, because when you have a car that can self update over the internet to resolve a hardware problem, and that doesn't cost anybody almost anything. People don't have to drive their cars into a repair shop. Tesla doesn't have to employ people to fix it. This is now a huge, huge, huge competitive advantage where kind of General Motors has to fix cars, these guys issue a software patch. And this is the flat factory thing, this is happening now. And at the same time, continuous delivery starts creating some very, very unusual feedbacks on other parts of the business as well. For example, about two years ago, I started working on a collaboration tool with some friends, and we started doing stuff the way we're used to doing it. So we started with continuous delivery from day one, we were packaging stuff, releasing them very quickly. And this is our user growth graph over the last two years. Watching spot is somewhere towards the beginning, there's a massive spike. Then there's another kind of spike in another spike over there, very, very close to the beginning. And then for the next two years, no spikes at all. What happened there is we appeared on Hacker News on the front page, then what happened there is we appeared on Reddit on the front page. And then what happened there is the official version one was reviewed by PC World. What happened for the other one and a half years is we tried as much as we could to get anybody to write about us and failed. Although we did a ton of really, really interesting things, what we learned is that when you do continuous delivery, you take the drama out of the process. There's nothing newsworthy to write about. There's kind of nothing big to announce. There's no marketing opportunities anymore and kind of our marketing strategy that was to get blogs and newspapers to write about us was completely destroyed by the fact that we took the drama out of delivery. So this is a really interesting aspect that I never expected because previously when we did stuff like that, you'd release every six months or every year, you announced version 3.0, people write about that because journalists like to kind of write about something that's newsworthy. Now when you say, oh, we've updated this feature, we've updated that feature, there's nothing, nothing to come out of that. So kind of, although we perceived continuous delivery as a purely technical thing, it had a massive side effect on our marketing. Now that's not necessarily horribly bad. It's just that it prevented us from doing one marketing thing. We had to learn about that the hard way. And we can say, oh, I'm stupid because I make mistakes every year and I'm kind of coming here to talk to you about this. But other companies do this as well. And they get very, very surprised by the marketing side effects of continuous delivery because that's not a purely technical thing. For example, I'm going back to Tesla in October last year. Tesla announced a fantastic new feature that they're starting to put in their cars. And kind of Tesla with the whole continuous delivery model is trying to challenge this perception of models for cars. Ford, General Motors, Mercedes, they have models, Tesla kind of has a model S and just kind of ships updates. So they start announcing that, you know, from tomorrow morning, all model S cars that come out will have a front facing radar, 12 sonar sensors, several cameras, and they'll have an autopilot that's going to automatically change lanes. They'll have a digital break that's going to spot when there's a dangerous situation and kind of prevent stuff like that. So this is something that's amazing. It's kind of a fantastic safety feature. It's a fantastic feature anyway. And everybody was very excited about that. Newspapers wrote about how shit scared about this because when you think about Tesla doing automatic navigation, radar stuff, sensor stuff, you kind of know what the end game is going to be. So where kind of, I wasn't the only one that kind of felt negative about this. There was a guy called Richard Wolpert in the States that woke up one day, read the news about this, and he just felt that the world was unfair. Everything went wrong that day because he just bought a model S car that didn't have all these nice shiny features. And here's Tesla announcing that from tomorrow everybody else is going to get the radar, a sonar, and I don't know why they need a sonar. I guess they're preparing a car to go underwater. But kind of, you know, 12 sonars, two radars, cameras, automatic navigation, things like that. Richard felt that the world was just unfair. So what do people do when they feel the world is unfair? They start bitching about it online. And he started a petition to kind of force Tesla to get autopilot radars and sonars retrofitted into all the old models. And 981, 948 people signed up for this petition to kind of start forcing Tesla to retrofit stuff because people felt entitled. As continuous delivery teaches people that we can now ship stuff easily, update stuff, users start feeling more entitled. Now, I've never heard of anybody going to Mercedes or to Ford and saying, Oh, you have a new model. I bought the car five years ago. I want everything that the new car has for free. But Tesla taught their users to expect more. And, you know, people all over the world signed up for this. Where, you know, even in Norway, there was a guy in Norway called Jochen, who said, Hey, because it's been said, I want this stuff, I'm entitled to this stuff. And this is because kind of when we as we start doing continuous delivery, our stakeholders, our users start expecting stuff that might be completely unrealistic. And, you know, installing radars and sonars and stuff like that, you know, that's not a software update thing. That's kind of Star Trek replicator stuff. That's not something you can do very, very easily. But again, these people felt entitled. So regardless of whether that's possible or not possible, disappointing users is never good. Making users angry enough to start an online petition against you is never good. Tesla is still in an early adopter stage. And at that point, they were selling something like one and a half thousand cars a month. So if you have 1000 people pissed off enough to sign an online petition, and you're selling one and a half thousand cars a month, that's not good. So kind of this is where I think we need to start thinking about this slightly differently. And especially when kind of users are concerned, continuous delivery is typically something that technical people approach as a way to reduce technical risk. And users typically get at the wrong end of the stick. So if you're doing continuous delivery, please, please, please, please, consider your poor users. For example, I tend to travel a lot for conferences. And I was in Australia last year. Australia, unlike kind of normal countries, has stupidly expensive internet. I'm talking about, say, $50 for three gigabytes. And they sell it by gigabyte, they don't sell it by time based access and things like that. So as we were there, the conference organizers gave us these small kind of my five devices to use so that we don't pay the hotels. And I kind of like to fiddle with my slides until the very last moment I was changing the slides as people were coming into this room now. So about two hours before my talk in Australia, I finally sat down to do my slides. And normally that's not a problem at all. You go to Google Images, you find Patrick Stewart, you put it on, and that's perfect. That day, the world seemed unfair. None of the browsers seem to work. Everything seemed to crash all the time. My ping was 10 seconds to anything. And I couldn't figure out what was going on. After about half an hour of panicking, restarting the computer, trying to figure out whether Apple's network demon is a fault or whatever is going on, I've realized that my iPad decided on its own to update iOS and kind of took out the complete pipe I had. And not only that, but kind of spent all the bandwidth I had left. So I had to pay $50 for two more gigabytes. Then I had to find slides in a panic mode and kind of do with a half-finished presentation. So fair enough, that's my fault. It's my fault for not disabling automatic updates. But at the same time, Apple had an important update. And the update is never, ever, ever going to be as important as the work I have to do now. When people have an important security update, that's important for the company doing the update. That's basically saying, we cocked up, we're embarrassed, and now we want to update your system. If I'm doing something at that moment, that's not important at all. That's very low on the scale of importance. Likewise, for the content managers at TV Station, the technical update that was done for some other users somewhere half way around the world was completely unimportant to them. What was important is they were half way through writing a story, and now they have to log out and re-log in again. And I think we need to start considering that if we're not going to piece of users, if you really, really, really, really, really, really, really want a piece of users, then what you should do is deploy UX changes partially. And do continuous delivery on the UX. For example, PayPal did this to me about six months ago. They were starting to deploy the new business dashboard. Whenever you hear news somewhere, run away, run to the hills. So I woke up one day and there were a few thousand dollars missing from my company PayPal account. Now, that's never good. And because several people have access to the account, I didn't know whether somebody took the money out or whether they were going to go to where the kind of we got defrauded or something like that. So in panic mode, I started downloading transactions, figuring out what was going on, adding things up, but then they just didn't add up. If I added up the transactions, it looked fine. If I looked at the balance in the business dashboard, several thousand dollars were missing. So kind of I noticed a link on top. This is switched to classic and I put my testers hat down and I thought somebody must have caught something up. So I switched to classic and there was the money. So what effectively they did is we had multiple currencies in our accounts. And because the primary account was British pounds, they kind of deployed the UX changes gradually and they thought, well, we're just going to do it first for people with one account. And they were only showing the British balance. Now, I could put my testers hat on and I could notice the switch to classic thing, but I can bet you lots and lots of people that they had a really bad day and the call center supported people had a really bad day as well. So kind of that's why we're, you know, one of the biggest problems when we talk about continuous delivery, although continuous delivery is incredibly empowering. It opens up new business opportunities. It can kind of help you be Tesla, not general motors and stuff like that. When you go and talk to people about doing big stuff, they're really, really scared. An insurance company I worked with, when they started moving to continuous delivery, the business users thought, this is really brilliant. This is great. We can do small stuff easily, but our technology lost the capability of doing any big changes anymore because they just fight anything that's big. When you go and talk to UX people and you say, okay, we're going to start updating one page and we're going to update another, they start running for the hills because if we start looking at this from a development perspective, we think what we're doing is Amazon Prime. We think we're shipping stuff well. We think it's all fine. It's all brilliant. But when you look at it from a user perspective, from a UX perspective, what those people feel like is they're being force fed. And there's kind of a massive difference between continuous delivery this way and continuous delivery that way. So one of the most important things we need to start thinking about, especially kind of doing big, big changes and not losing the capability of doing big stuff is not doing this stuff over here. So kind of, I've almost always, when we start talking about continuous delivery to business people in terms of reducing technical risk, they love it. Then you go and talk to kind of how you're going to do a big technology refresh or how you're going to risk in the website, how you're going to rebrand something when you need, and people start fighting you with everything they can. UX designers kind of complain that you can't go and confuse users because if you start rebranding one part of the website, the other part is going to be completely confused. And that's what happened to us. I mean, I can't complain about that. And especially when you start thinking about slicing things, it's almost impossible to slice continuous delivery of UX in any sensible way. People will always be confused and it's always going to be stupid. So we can either not do continuous delivery or we can do it in a way that confuses people. And that's where kind of, this is really, really a big challenge. So one of the most important things to unlock the potential of this over the next five years is going to look at how do we not lose human control over this? How do we not go for false feeding? How do we go for Amazon Prime? And how do we, if we have to do this, because we'll have to do it. But how do we at least keep human control over that? So we do it in a sensible way because there are sensible ways of doing this. So kind of going back to the three rules of robotics, I think kind of what I'd like to rewrite this, so we're not going to talk about robots anymore, we'll talk about continuous delivery. And the first rule that talks about may not injure a human. Fair enough, if people are doing kind of medical devices, I hope you already have this in your testing. But when we talk about continuous delivery, particularly when we talk about UX, I think the biggest problem here is confusion. So the first rule of continuous delivery that we really have to put in place is this can't confuse users. Now, the second rule talks about obeying orders for humans, unless kind of order one, and I'm going to kind of skip the unless order, unless rule one. I think kind of this thing may must obey users when we think about what people actually want in terms of continuous delivery. Most often, they don't want it. If I'm in the middle of something, I do not want the ground to move. I want the stuff to stay the same. And I want this stuff, if it has to happen, to happen without me noticing at all. So the second rule of continuous delivery, if we're going to do it right, I'm going to replace must obey orders from must not interrupt users' work or sessions, because that's the really, really important thing. If that happens, I don't care about. Now, the third one is interesting kind of must protect its own existence. And where continuous delivery fails to protect its own existence, if it starts fighting the business. If there's a business reason why we want to risk in the website, or there's a business reason why we want to do technology refresh, we want to move from Oracle to in-memory data grids, we want to do a huge change that is going to involve rewriting everything to work on a mobile and things like that. Continuous delivery cannot fight the business there. If we start saying, oh, you know, that's possible, but only if we start moving people and for a while everything's going to be messed up, that's not going to work. And one of the most important things there is, I think, continuous delivery has a huge potential to disrupt marketing. That's what we felt on our own scale. That's what Tesla felt. So I think kind of continuous delivery rule number three in terms of protecting its own existence must be not to disrupt or prevent marketing initiatives. And if we can do this kind of stuff, then we can put human control around all this robotic mechanism and we can actually benefit quite a lot from the whole thing. So when we started looking at these three things for the collaboration tool I mentioned, we realized very, very quickly that we had a fundamental misunderstanding of two really important concepts. In our mind, deployment and release were the same thing. And for most software people, and it's likely that most of in the audience are now kind of scratching your heads, deployment and release are the same thing. But kind of we realized that's completely wrong. And that's one of those constraints of the old world. That's a tall factory. And if we can move away from that because those constraints no longer apply, then we can get really, really, really, really interesting benefits out of the whole thing. So the first thing that I'd really like you to kind of remember from this is that deployment isn't and shouldn't be the same thing as a release. Deployment is a technical event of pushing software to production. And I'm really thinking production here. I don't care about deployment to staging. I don't care about deployment to development environments. That's all nice and fine. But deployment is deployment to production. That's a technical event of moving code to production. Now, most of you, I assume, have been kids at some point of your life. Some of you might even have kids. And people in those two categories know that the technical event of getting a package and the marketing event of making the package available to people is a completely different thing. So release is a marketing thing. That's what we realized. Release is a marketing event. It's a marketing decision. It's something that is done at the point where marketing people want to achieve the biggest impact. You hide your packages nicely in a closet so when Christmas comes, children get surprised. Where we need to start doing the same stuff with software. We can decouple the technical risk of deploying to production through continuous delivery. And we can enable a marketing decision of releasing completely separately. Now, the key thing to unlock that. And that's a really, really interesting thing to think about here because it took us a while to figure out how to do this. But there's one technical problem with this. And once that's solved, everything works okay. And it's not an easy technical problem. It's a very difficult technical problem. That's why I suggested it's going to take five years. So the key technical thing here is to support multiple versions of software in production at the same time. So we can deploy one thing. The other thing can keep running. That means that if we start thinking about interrupting people's sessions, not a problem at all. They can keep running whatever version of software they're running at the moment. If we start thinking about surprising people and confusing people with UX, that's not a problem at all. We can still keep working on this testing, deploying. If we talk about interrupting marketing initiatives or disrupting stuff, that's not a problem at all because marketing people can decide when to do a release. So that's kind of the key thing to solve. So for us, the biggest challenge when we looked at this kind of collaboration tool, restructuring it for that was data. And I think data is going to be the biggest problem for most people because you can pretty much run multiple versions of APIs. You can run multiple versions of sites, websites, scripts. That's not a problem. Data tends to be the biggest problem. So our approach to this was we slowly started putting in data versions on pretty much every single data item. And the biggest challenge for us was we don't own the data at all. Most data sits on people's computers or on their Google Drive or their Dropbox. So as we upgrade versions, I don't even have access to all the data that's out there. And I can't even guarantee that all the data is going to be upgraded over a period of time. We can get hit by some data item that's over a year old because people just didn't need to use that. And we started putting kind of data flags, version flags on each data item and making sure that kind of newer versions of software can always automatically upgrade any previous version. That's a nice technical problem. It's a solvable technical problem. It's something that you can unit test. It's kind of something that is a technical problem. We also made sure that each kind of version of software deploying is just going to ignore any data items it doesn't recognize. It's not going to delete them. So pretty much at this point, we were able to have two users collaborating on the same document, running with completely different versions of software where the person on the latest version will have all the brand new shiny features and they would see that in their data. The person on the old version will see what they can see with their version, but they're not going to be interrupted. They're going to use everything they could. They will not notice anything that happened until they decide to kind of update or refresh the page. So in terms of this stuff, when we started doing it, then we also started time-stumping and data versioning all the scripts. So pretty much everything that goes online is time-stumped when it went online and put on a CDN. So what we've achieved with this is if you're a user of a site and you hit a website, you get an HTML page and that HTML page has time-stumped JavaScript, time-stumped CSS, every other resource time-stumped. It knows the data version. It knows the API version. So as it sends stuff to APIs, it sends the version and until you refresh the page, you have no idea what we are doing in the background because you are loading scripts that already exist. You can kind of have a cached page, live with the cached page as much as you want. We are kind of deleting these files once every six months. And the odd thing about this is once we started doing this, we started realizing really, really weird stuff because I originally thought we'll only need to keep two versions, the previous one and the one that we're running now, but then we realized lots of mobile networks all over the world cache HTML files. They don't cache JavaScript. So people would get all the HTML files for a week. I have no idea where from. And they would ask for JavaScripts that should have been inaccessible for a week. But because we're doing this stuff, those people could still use their software. And these were some kind of really weird errors that we were getting in our logs. We couldn't understand once we started doing this, they were no longer a problem. So this kind of started opening up some really, really interesting opportunities for us, but kind of the really important thing that I want you to remember from this. And it's a technical problem to be solved. It's not an easy problem, but I think continuous delivery without multi version is just irresponsible. People that say that they're doing continuous delivery and they can't run multiple versions of software in production at the same time are damaging their business. That's it. So if we are going to be professionals, we need to start restructuring our systems, restructuring data, restructuring APIs to be able to do multi version stuff. Now, once we started doing this, the whole kind of tall factory thing broke down. And a completely new world opened up because all of the sudden, we could take all those big, massive things that were impossible to slice in any meaningful way. And they were actually quite easy to slice and push out. So one thing that we started looking at is now we had this stuff we could actually slice by, slice by capacity. Because we could start rolling stuff out that supports up to 10 users at a time. It's perfectly fine that we have hundreds and hundreds of users on the system at the moment. I think at our busiest moment, we had about 600 users concurrently. It's perfectly fine to do something that only supports 10. Because we can run multiple versions at the same time and we can deploy a version that supports only 10 users to up to 10 users at a time. And then test that, see how that's going. And if it's going well, start kind of upgrading the stuff in the back end to support 50 users at a time, then support 100 users at a time. And when you start doing stuff, like lots of really interesting opportunities open up, for example, you don't have to use the database, you can use a file. You don't have to support, you can support one user at a time, if you want. And things like this kind of really, really weird and interesting stuff open up that are now a possibility to slice things differently. And we can actually push this stuff to production to measure the impact of that on one user. But hey, we can at least prove what's happening. So the other thing that we kind of were able to do now is start slicing things that are big, big UX changes in a way that's a lot more meaningful. For example, from Chrome version 26 to Chrome version 34, about four versions came up with broken canvas. That was a big problem for us because our key platform for visualization was canvas. And when Chrome breaks canvas, about 50% of our users pretty much complain to us. And we got a ton of angry emails all the time. And that's not something I can fix. I have no access to Chrome code. I wouldn't know even where to fix it. And generally kind of the Chrome guys were depending on where you look at it, either relatively good fixing it because it only took them a couple of days or really, really slow fixing it because it only took them a couple of days. But we had outages for 50% of our users for a couple of days where in some situations we could provide workarounds. Some situations, no way to do anything. Like one time we discovered that if you disable 3D acceleration, things work okay. So we sent emails to people. But then that's not really the best because they've disabled it globally. Some other changes just we couldn't fix. And we decided that kind of canvas is too unstable people are kind of still developing it. We're going to move to SVG. Now, this is one of those typical things where it's all or nothing because the value only comes where you can kill canvas completely. And if you're doing stuff gradually say you keep people both on canvas and SVG, that doesn't make sense at all. Because they're still going to be heard by canvas and your half baked SVG implementation is going to be ridiculous. So kind of no point at all until you have a big switch. But what we started thinking about is, hey, maybe we don't need to provide a whole implementation. Maybe we can look at getting 1% of our users on SVG and giving them an incentive to move to that. So we started looking at slicing by capacity and we did, oh, you know, our documents can have thousands and thousands and thousands of entries. The first thing we're going to do is we're going to do SVG that's not performance optimized. So we're going to support up to 100 elements. And we're not going to support all the features. We're going to support only these things. But what we're going to do is we're going to give people a nice incentive to choose this new stuff. So we deployed a very slimmed down version of SVG support that was performance wise bad that didn't have all the features, but it was enough for a sub segment of users to say, hey, this is good. I'm going to try this. And unlike PayPal that forced people to use the new business dashboard, what we did is we also introduced something that was new and funky. And we invited people to turn it on. So we gave people some new customizations and we said, look, you know, if you try this now, switch to it and you can get this fully customizable visuals. The performance is not good. You're not going to get all the features, blah, blah, blah, blah, blah, but try it out. It might work for you. And if it doesn't, you can always switch back. And by doing that, we're able to gradually kind of move people on. We were able to test that SVG is actually good. And during this kind of two months move to SVG, Chrome actually came up with a broken canvas. At this point, we were able to do about 50% of the users and just said, look, by now, we're just going to switch people to default SVG. If you're having problems, move back to canvas. But because Chrome is kind of borked, it might not work. And this is what Marty Kagan describes as gentle deployments in his book, Inspired. He talks about doing big exchanges by releasing something to a small portion of your users, inviting them to try it out and giving them the benefit to switch to the new thing while still allowing people to switch back to the old version if they have problems. This makes you exchanges zero risk because people intentionally switching to it, expecting consistency, so they're not going to be confused. They know what they're doing. They know why they're doing it. And if they have any problems, they can switch back. So that's a really nice way of kind of testing you exchanges because we can start bringing users gradually. We can start testing that those things make sense. And we can start doing technology refreshes that way. We can start moving people gradually to that. So kind of the key thing that I kind of like you to think about here is that kind of generally two things to remember from this are decouple deployment and release. Deployment is a technical event. Release is a marketing event. Although we were trained to think that those two things are the same, they're actually not. The tight coupling of deployment and release is a constraint of the old world. That's kind of the tall factory thing. And what we need to do is move away from that. And the kind of the right way to start doing that is to start supporting multi-version in production. And that's a big technical challenge for a lot of people, but it's a technical problem. It's something we can solve. And then we can reduce the risk of business changes, big UX refreshes, big technology refreshes, reducing kind of those big risks down to a technical problem. That's amazing. The second thing is this allows us now to start planning to give 2% of our users 100% of what they need rather than force feeding people with what they don't really want. And I think that's kind of the new world. That's where we start breaking the constraints of the old world because we can start doing that actively in production. And kind of when we start doing this amazing stuff start to happen. For example, there's a fantastic episode from Google in 2009 that is known as 41 Shades of Blue. This came up five years before the 50 Shades of Grace. So kind of, you know, they were there first and I think they could patent the whole thing. So at some point, Google's director of design had an idea to change the color of the hyperlinks in Google Ads on the homepage. And he proposed that they need to change this to this color because it's much more appealing to human eye. It's going to be great. It's going to be fantastic. The design is going to be much better. And because the designers were a bit pissed off at him for kind of previous stuff, they didn't just jump into doing this. They approached this as, okay, what we're going to do is we're going to decouple deployment and release. And we're going to scale this out so that marketing people can release it as they want. And by the way, because we don't trust that this is a good idea, what we're going to do is we're going to do the same thing with 40 other Shades of Blue at the same time. And we're going to let the marketing people decide on the release. We've done our technical part. Now other people can play with their stuff. And what they've done is kind of the marketing guys tested this on 1% of their users. And they proved that the color that the head of design suggested was actually performing much worse than the current one. And multiplied to 100% of the users and a year, the estimate was that the change in click rates would be about $200 million of damage. Now, instead of just saying, oh, you know, we got this requirement, we can roll it out quickly, we've rolled it out, it's nice, it's brilliant, we do continuous delivery, these guys decouple deployment and release. And as a result, they saved $200 million. The head of design actually quit as a result of that. So they saved $200 million plus a salary of an overpaid designer. And he went on to work for Twitter where they're still trying to figure out how to make money. So kind of that's the kind of stuff we can start doing if we do those two things. And they're not going to be easy, but they're incredibly powerful. So that's my message to you. Those are the two things to remember now. Kind of you had a good idea about the kind of three rules. And I think you said something. So you get two more books, I have two books here. The others, I don't want to make you feel bad, because you know, now you feel entitled. So I'll give you an electronic copy of this book. So if you go to this URL over the next month, you can download any copy of the book, it automatically reduces the price to zero. And a lot of the ideas that I've mentioned today are explained there in a bit more detail with some more references for research. So that's pretty much it. I hope I've challenged your thinking at least a bit. And I have about five minutes for questions, but then I have to go because I'll miss my plane. Does anybody have any questions that are kind of burning and you're not going to be able to sleep? Good. Okay, I'll go and catch my plane then. Thank you very much. Okay.
a lot more important in your company. And faster and faster and faster, they start thinking the business is a shiny new toy that's really the industry, especially over the last five or thing, this is happening now. And at the same potential of this over the next five years is testing. But when we talk about continuous people and for a while everything's going to be and we can actually benefit quite a lot from the thing to unlock that. And that's a really, really will have all the brand new shiny features and you get an HTML page and that HTML page has going to be much better. And because the